> ## Documentation Index
> Fetch the complete documentation index at: https://ulpi.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# How It Works

> The AI magic behind semantic search: How ULPI turns scattered docs into instant answers

# The Technology Behind Instant Documentation Search

**Ever wonder how ULPI makes your AI instantly know your entire codebase?**

It's not magic—it's a carefully engineered pipeline that transforms scattered documentation into semantic search that actually works.

**This guide explains:**

* 🔍 How semantic search understands meaning, not just keywords
* ⚡ Why ULPI is 25x more efficient than loading full docs
* 🔄 How updates sync in under 60 seconds
* 🔒 How your private docs stay secure

<Note>
  **Not interested in technical details?** Skip to [Search Features](/documentation/search-features) for practical usage tips.

  **Want to integrate via API?** Jump to [API Integration](/documentation/api-integration).
</Note>

***

## The Problem: Why Keyword Search Fails for Docs

**Traditional search tools** (GitHub search, grep, etc.) look for exact keyword matches:

<Tabs>
  <Tab title="Keyword Search Fails">
    **You ask:** "How do I handle database schema changes?"

    **Keyword search thinks:**

    ```
    Find files containing:
    - "database" AND "schema" AND "changes"
    ```

    **Results:**

    * ❌ Misses `docs/migrations.md` (doesn't mention "schema changes")
    * ❌ Misses `architecture/versioning.md` (doesn't say "database")
    * ✅ Finds `README.md` with phrase "database schema changes" (lucky!)

    **Problem:** Your migrations guide uses words like "migration" and "versioning"—not "schema changes"
  </Tab>

  <Tab title="Semantic Search Succeeds">
    **You ask:** "How do I handle database schema changes?"

    **Semantic search thinks:**

    ```
    Find documents about:
    - Database migrations
    - Schema versioning
    - DDL changes
    - Database upgrades
    - Data structure evolution
    ```

    **Results:**

    * ✅ Finds `docs/migrations.md` (about database migrations)
    * ✅ Finds `architecture/versioning.md` (explains schema evolution)
    * ✅ Finds `backend/README.md#migrations` (even without exact keywords)

    **Why it works:** Understands that "schema changes" = "migrations" = "versioning"
  </Tab>
</Tabs>

**This is why ULPI uses semantic search powered by vector embeddings.**

***

## High-Level: How It Works in 30 Seconds

<Steps>
  <Step title="You Connect Repositories (2 minutes)">
    **One-click OAuth connection** to GitHub, GitLab, Bitbucket, or Gitea

    ULPI automatically discovers all your documentation:

    * README files
    * `docs/` directories
    * Wikis
    * Markdown files everywhere
  </Step>

  <Step title="AI Indexes Your Docs (2-5 minutes)">
    **Automatic processing in the background:**

    1. Parse every documentation file
    2. Break into logical sections (chunking)
    3. Convert to vector embeddings (AI representation)
    4. Store in lightning-fast search engine (Typesense)

    **You don't do anything.** It just works.
  </Step>

  <Step title="Your AI Asks Questions (instant)">
    **When your AI assistant needs docs:**

    ```
    AI: "I need docs about authentication"
    ↓
    ULPI: Searches all repos semantically (40ms)
    ↓
    ULPI: Returns relevant sections (2,000 tokens)
    ↓
    AI: "Based on your docs/auth.md, here's how..."
    ```

    **Total time:** Under 50 milliseconds
  </Step>

  <Step title="Auto-Sync on Every Push (60 seconds)">
    **You push to main:**

    ```bash theme={null}
    git push origin main
    ```

    **ULPI automatically:**

    1. Receives webhook notification (1 second)
    2. Re-indexes changed files (30-60 seconds)
    3. AI now sees updated documentation

    **No manual sync button.** Always up-to-date.
  </Step>
</Steps>

<Info>
  **The magic:** AI understands *meaning*, not just keywords. That's why it finds the right docs even when you use different terminology.
</Info>

***

## Deep Dive: The Indexing Pipeline

**For developers who want to understand the technical implementation.**

### Architecture Diagram

```mermaid theme={null}
sequenceDiagram
    participant User as Your Git Repo
    participant Webhook as GitHub Webhook
    participant API as ULPI API
    participant Queue as Processing Queue
    participant Embeddings as OpenAI Embeddings
    participant Vector as Typesense Vector DB
    participant MCP as MCP Server
    participant AI as Your AI Assistant

    User->>Webhook: git push
    Webhook->>API: Push event
    API->>Queue: Queue indexing job
    Queue->>API: Fetch changed files
    API->>Embeddings: Generate embeddings
    Embeddings-->>API: Vector embeddings
    API->>Vector: Store embeddings

    AI->>MCP: Search "authentication"
    MCP->>API: Search request
    API->>Vector: Vector similarity search
    Vector-->>API: Top 5 matches
    API-->>MCP: Search results
    MCP-->>AI: Documentation context
```

### Step-by-Step: What Happens During Indexing

<AccordionGroup>
  <Accordion title="Step 1: Repository Connection & Discovery" icon="link">
    **When you connect a repository:**

    <Steps>
      <Step title="OAuth Authentication">
        **Read-only access** via GitHub/GitLab OAuth

        **Permissions requested:**

        * ✅ Read repository contents
        * ✅ Register webhooks for auto-sync
        * ❌ **No write access** (we never modify your code)
      </Step>

      <Step title="File Discovery">
        ULPI scans your repository structure:

        ```
        your-repo/
        ├── README.md                    ✅ Indexed
        ├── docs/
        │   ├── setup.md                 ✅ Indexed
        │   ├── api-reference.md         ✅ Indexed
        │   └── architecture.md          ✅ Indexed
        ├── .github/
        │   ├── CONTRIBUTING.md          ✅ Indexed
        │   └── workflows/deploy.yml     ❌ Not docs
        ├── src/
        │   ├── UserService.php          ❌ Code (not indexed by default)
        │   └── README.md                ✅ Indexed (README)
        ├── node_modules/                ❌ Excluded
        └── vendor/                      ❌ Excluded
        ```

        **Indexed file types:**

        * `.md`, `.mdx` (Markdown)
        * `.txt` (plain text in doc directories)
        * `README.*` (any extension, any directory)
        * Wiki pages (optional)

        **Custom exclusions:** Add `.ulpiignore` file
      </Step>

      <Step title="Metadata Extraction">
        For each discovered file, ULPI extracts:

        * **Path:** `docs/authentication.md`
        * **Branch:** `main`
        * **Last modified:** Git commit timestamp
        * **Author:** Git commit author
        * **File size:** For processing estimates
      </Step>
    </Steps>
  </Accordion>

  <Accordion title="Step 2: Content Parsing & Chunking" icon="scissors">
    **Breaking documents into searchable chunks**

    ### Why Chunking?

    **Problem:** A 10,000-line architecture doc contains dozens of distinct topics.

    **Without chunking:**

    * Search returns entire 10,000-line document
    * AI must process all 10,000 lines to find relevant section
    * Wastes 8,000 tokens on irrelevant content

    **With chunking:**

    * Search returns only the relevant 200-line section about your query
    * AI gets precise context
    * Saves 97% of tokens

    ### How We Chunk

    **Smart chunking strategy** (not just splitting every N characters):

    1. **Respect document structure:**
       * Preserve headings and sections
       * Keep code blocks together
       * Don't split tables or lists

    2. **Optimal chunk size: \~512 tokens**
       * Large enough for context
       * Small enough for precision
       * Equivalent to 2-3 paragraphs

    3. **Add overlap:**
       * 50-token overlap between chunks
       * Prevents losing context at chunk boundaries

    **Example:**

    ```markdown theme={null}
    ## Authentication

    Our API uses OAuth 2.0 for authentication...
    [500 tokens about OAuth]

    ─────── CHUNK BOUNDARY with 50-token overlap ───────

    ## Authorization

    After authentication, users are assigned roles...
    [500 tokens about authorization]
    ```

    **Result:** When searching for "authentication", you get the OAuth section—not the entire auth guide.
  </Accordion>

  <Accordion title="Step 3: Vector Embedding Generation" icon="brain">
    **Converting text to AI-understandable format**

    ### What Are Embeddings?

    **Simple explanation:** Embeddings convert text into numbers that capture meaning.

    **Example:**

    ```
    Text: "How to authenticate users with OAuth"
    ↓
    Embedding: [0.023, -0.891, 0.456, 0.123, ... 1,536 numbers total]
    ```

    **Why numbers?** Computers can compare numbers to find similar meanings:

    ```
    "authenticate users"     → [0.23, -0.89, 0.45, ...]
    "login system"           → [0.21, -0.87, 0.44, ...]  ← Very similar!
    "database migrations"    → [0.87, 0.23, -0.12, ...] ← Very different
    ```

    ### The Process

    <Steps>
      <Step title="Send to OpenAI">
        Each chunk is sent to OpenAI's embedding API:

        **Model:** `text-embedding-3-large`

        **Dimensions:** 1,536 (captures nuanced meaning)

        ```typescript theme={null}
        const embedding = await openai.embeddings.create({
          model: "text-embedding-3-large",
          input: "Our API uses OAuth 2.0 for authentication...",
        });

        // Returns: [0.023, -0.891, 0.456, ..., 1536 numbers]
        ```
      </Step>

      <Step title="Store in Vector Database">
        Embeddings are stored in Typesense:

        ```json theme={null}
        {
          "id": "doc_123_chunk_4",
          "content": "Our API uses OAuth 2.0 for authentication...",
          "embedding": [0.023, -0.891, 0.456, ...],
          "metadata": {
            "file": "docs/authentication.md",
            "repository": "backend-api",
            "branch": "main",
            "url": "github.com/org/repo/blob/main/docs/auth.md#L42"
          }
        }
        ```
      </Step>
    </Steps>

    ### Privacy Note

    <Warning>
      **Your documentation is NEVER used to train AI models.**

      * Embeddings are mathematical representations, not readable text
      * OpenAI's zero data retention policy (embeddings API)
      * We only use embeddings for search indexing
      * Original text stays in your control
    </Warning>
  </Accordion>

  <Accordion title="Step 4: Indexing in Typesense" icon="database">
    **Storing embeddings for lightning-fast search**

    ### Why Typesense?

    **Typesense** is an open-source vector search engine optimized for:

    * ⚡ **Speed:** Sub-50ms vector similarity search
    * 🎯 **Accuracy:** Hybrid semantic + keyword ranking
    * 📈 **Scalability:** Handles millions of documents
    * 🔧 **Simplicity:** No complex configuration

    ### Index Structure

    ```typescript theme={null}
    {
      "name": "documentation",
      "fields": [
        { "name": "content", "type": "string" },
        { "name": "embedding", "type": "float[]", "num_dim": 1536 },
        { "name": "file_path", "type": "string", "facet": true },
        { "name": "repository", "type": "string", "facet": true },
        { "name": "branch", "type": "string", "facet": true },
        { "name": "last_modified", "type": "int64", "sort": true }
      ]
    }
    ```

    **Facets:** Enable filtering by repository, file type, branch, etc.
    **Sorting:** Prioritize recent documentation
  </Accordion>

  <Accordion title="Step 5: Webhook Registration" icon="bell">
    **Auto-sync on every git push**

    ### How Webhooks Work

    When you connect a repository, ULPI registers a webhook:

    **GitHub webhook payload:**

    ```json theme={null}
    {
      "ref": "refs/heads/main",
      "commits": [
        {
          "modified": ["docs/authentication.md"],
          "added": ["docs/new-feature.md"],
          "removed": ["docs/deprecated.md"]
        }
      ]
    }
    ```

    **ULPI receives this and:**

    1. Queues a re-indexing job
    2. Fetches only changed files
    3. Re-generates embeddings for changes
    4. Updates Typesense index
    5. Invalidates cache

    **Time:** 10-60 seconds from push to searchable

    ### Smart Re-indexing

    **Full re-index (slow):**

    * New documentation directory created
    * Branch created/deleted
    * Manual trigger from dashboard

    **Partial re-index (fast):**

    * Existing file modified
    * Only re-process changed chunks
    * **10-30 seconds**

    **No re-index:**

    * Code files changed (`.js`, `.php`, etc.)
    * Excluded directories (`node_modules/`, `vendor/`)
    * Files in `.ulpiignore`
  </Accordion>
</AccordionGroup>

***

## How Semantic Search Works

**When your AI assistant queries ULPI:**

### The Search Process

<Steps>
  <Step title="Query Embedding">
    **Your AI asks:** "How do I deploy to production?"

    **ULPI converts to embedding:**

    ```typescript theme={null}
    const queryEmbedding = await openai.embeddings.create({
      model: "text-embedding-3-large",
      input: "How do I deploy to production?"
    });

    // Result: [0.123, -0.456, 0.789, ..., 1536 dimensions]
    ```

    **Same model** as document embeddings → comparable vectors
  </Step>

  <Step title="Vector Similarity Search">
    **Find similar documentation chunks:**

    **Algorithm:** Cosine similarity

    ```
    similarity = cos(query_vector, doc_vector)

    Range: 0.0 (completely different) to 1.0 (identical)
    ```

    **Typesense finds:**

    ```json theme={null}
    [
      {
        "content": "Deploy to production using...",
        "similarity": 0.94,
        "file": "infrastructure/deployment.md"
      },
      {
        "content": "Our deployment process involves...",
        "similarity": 0.89,
        "file": "docs/ci-cd.md"
      },
      {
        "content": "Production environment configuration...",
        "similarity": 0.85,
        "file": "README.md"
      }
    ]
    ```

    **Threshold:** Returns matches with similarity > 0.75
  </Step>

  <Step title="Hybrid Scoring">
    **Combine semantic + keyword matching:**

    ```typescript theme={null}
    final_score = (0.7 * semantic_score) + (0.3 * keyword_score)
    ```

    **Why hybrid?**

    <Tabs>
      <Tab title="Semantic Only (Problem)">
        **Query:** "Redis configuration"

        **Semantic search finds:**

        * ✅ "Caching setup" (semantically similar)
        * ✅ "Session storage" (related concept)
        * ❌ Misses exact "redis.conf" reference

        **Issue:** May miss exact keyword matches
      </Tab>

      <Tab title="Keyword Only (Problem)">
        **Query:** "Redis configuration"

        **Keyword search finds:**

        * ✅ "redis.conf" (exact match)
        * ❌ Misses "caching setup" (no keyword "redis")
        * ❌ Misses "session storage" (no keyword "redis")

        **Issue:** Misses related concepts
      </Tab>

      <Tab title="Hybrid (Best)">
        **Query:** "Redis configuration"

        **Hybrid search finds:**

        * ✅ "redis.conf" (keyword match + somewhat semantic)
        * ✅ "Caching setup with Redis" (both)
        * ✅ "Session storage using cache" (semantic)

        **Result:** Gets both exact matches AND related concepts
      </Tab>
    </Tabs>
  </Step>

  <Step title="Ranking & Filtering">
    **Results are ranked by:**

    1. **Relevance score** (hybrid score)
    2. **Document recency** (newer docs ranked higher)
    3. **Repository priority** (if you specified repos)
    4. **File type:**
       * `README.md` (most important)
       * `docs/*.md` (documentation)
       * Other files (lower priority)

    **Filters applied:**

    * Repository scope (if API key is scoped)
    * Branch (default: `main`)
    * File type (if specified)
    * Date range (if specified)
  </Step>

  <Step title="Context Assembly">
    **Return top results with metadata:**

    ```json theme={null}
    {
      "results": [
        {
          "content": "Deploy to production using GitHub Actions...",
          "file": "infrastructure/deployment.md",
          "repository": "backend-api",
          "branch": "main",
          "url": "https://github.com/org/backend-api/blob/main/infrastructure/deployment.md#L42-L67",
          "score": 0.94,
          "last_modified": "2025-11-10T14:23:00Z"
        }
      ],
      "total": 12,
      "tokens_used": 2340
    }
    ```

    **AI receives this and synthesizes answer:**

    ```
    "Based on your deployment guide in infrastructure/deployment.md:

    To deploy to production:
    1. Run `npm run build`
    2. Push to deploy branch: `git push origin main:deploy`
    3. GitHub Actions automatically deploys to AWS

    [Link to deployment.md:42-67]"
    ```
  </Step>
</Steps>

### Why This Is Fast

**Sub-50ms average latency:**

<CardGroup cols={2}>
  <Card title="Optimized Vector Search" icon="gauge-high" color="#10b981">
    Typesense pre-computes indexes for instant similarity search

    **No full-text scanning required**
  </Card>

  <Card title="Three-Level Caching" icon="bolt" color="#3b82f6">
    * Browser cache (5 min)
    * MCP server cache (1 hour)
    * API cache (5 min)

    **Repeated queries: 5ms**
  </Card>

  <Card title="Smart Chunking" icon="scissors" color="#8b5cf6">
    Returns only relevant sections, not entire files

    **2,000 tokens vs 50,000 tokens**
  </Card>

  <Card title="Dedicated Infrastructure" icon="server" color="#f59e0b">
    Separate Typesense cluster for search

    **No database bottlenecks**
  </Card>
</CardGroup>

***

## Real-Time Updates via Webhooks

**How documentation stays synchronized automatically**

### Webhook Flow

```mermaid theme={null}
sequenceDiagram
    participant Dev as Developer
    participant Git as GitHub/GitLab
    participant Webhook as ULPI Webhook Endpoint
    participant Queue as Laravel Queue (Redis)
    participant Worker as Queue Worker
    participant Typesense as Typesense Index

    Dev->>Git: git push origin main
    Git->>Webhook: POST /webhooks/github
    Webhook->>Queue: Dispatch IndexDocumentationJob
    Webhook-->>Git: 200 OK (immediate response)

    Queue->>Worker: Process job
    Worker->>Git: Fetch changed files
    Git-->>Worker: File contents
    Worker->>Worker: Generate embeddings
    Worker->>Typesense: Update index
    Typesense-->>Worker: Indexed
    Worker->>Worker: Invalidate cache
```

### Processing Timeline

**What happens after you push:**

<Steps>
  <Step title="Instant: Webhook Received">
    **\< 1 second**

    GitHub/GitLab sends webhook to ULPI:

    ```json theme={null}
    POST /webhooks/github
    {
      "repository": "your-org/backend-api",
      "ref": "refs/heads/main",
      "commits": [
        {
          "modified": ["docs/authentication.md"],
          "added": ["docs/new-api.md"]
        }
      ]
    }
    ```

    ULPI responds **200 OK** immediately (non-blocking)
  </Step>

  <Step title="5-10 seconds: Job Queued">
    **Background processing starts:**

    1. Job added to Redis queue
    2. Laravel Horizon assigns worker
    3. Worker fetches changed files from GitHub
  </Step>

  <Step title="20-40 seconds: Re-indexing">
    **For each changed file:**

    1. Parse Markdown content
    2. Chunk into sections
    3. Generate embeddings (OpenAI API call)
    4. Update Typesense index
    5. Invalidate cached searches
  </Step>

  <Step title="30-60 seconds: Searchable">
    **Documentation is now searchable:**

    * ✅ AI assistants see updated docs
    * ✅ New sections discoverable
    * ✅ Deleted sections removed
    * ✅ Cache cleared
  </Step>
</Steps>

**Total time: 30-60 seconds** from push to searchable

<Info>
  **Large pushes (100+ files):** May take 2-5 minutes. Check indexing status in dashboard.
</Info>

***

## MCP Integration Architecture

**How AI assistants access your documentation**

### MCP Protocol Overview

**MCP (Model Context Protocol)** is a standard for connecting AI tools to external data sources.

**ULPI provides an MCP server** that bridges AI assistants to the ULPI API:

```
┌─────────────────┐
│  Your AI        │  (Claude Desktop, Cursor, etc.)
│  Assistant      │
└────────┬────────┘
         │
         │ MCP Protocol
         │
┌────────▼────────┐
│  ULPI MCP       │  (@ulpi/mcp-server-documentation)
│  Server         │  - Runs locally in your IDE
│                 │  - Authenticates with API key
└────────┬────────┘
         │
         │ HTTPS
         │
┌────────▼────────┐
│  ULPI API       │  (api.ulpi.io)
│  - Search       │  - Hosted in AWS
│  - Indexing     │  - Handles all teams
└─────────────────┘
```

### MCP Server Implementation

**The MCP server provides tools to your AI:**

```typescript theme={null}
// @ulpi/mcp-server-documentation

const tools = [
  {
    name: "search_documentation",
    description: "Search all connected repositories for documentation",
    parameters: {
      query: "Natural language search query",
      repository: "Optional: Filter to specific repo",
      limit: "Max results to return (default: 5)"
    }
  }
];

// When AI calls the tool:
async function search_documentation({ query, repository, limit = 5 }) {
  // 1. Authenticate with API key from environment
  const apiKey = process.env.ULPI_API_KEY;

  // 2. Call ULPI API
  const response = await fetch('https://api.ulpi.io/api/v1/documentation/search', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ query, repository, limit })
  });

  // 3. Return results to AI
  return await response.json();
}
```

### AI Tool Invocation Example

**How Claude Desktop uses ULPI MCP:**

```
User: "How do we handle authentication in the API?"

Claude (thinking):
  I need to search the user's documentation for authentication info.
  I'll use the search_documentation tool.

Claude → MCP Server:
  search_documentation({
    query: "API authentication implementation",
    limit: 5
  })

MCP Server → ULPI API:
  POST /api/v1/documentation/search
  {
    "query": "API authentication implementation",
    "limit": 5
  }

ULPI API → MCP Server:
  {
    "results": [
      {
        "content": "Our API uses Laravel Sanctum for authentication...",
        "file": "backend-api/docs/auth.md",
        "url": "github.com/org/backend/blob/main/docs/auth.md#L15"
      }
    ]
  }

MCP Server → Claude:
  [Returns search results]

Claude → User:
  "Based on your documentation in backend-api/docs/auth.md, your API uses
   Laravel Sanctum for authentication. Here's how it works:

   1. Client sends email + password to /api/login
   2. Server validates credentials
   3. Sanctum generates API token
   4. Client includes token in Authorization header

   [Link to auth.md:15-42]"
```

**Seamless.** User doesn't see any of this—just accurate answers from their docs.

***

## Caching Strategy

**Three-level cache for optimal performance**

<Tabs>
  <Tab title="Level 1: Browser/IDE Cache">
    **Location:** User's browser or IDE

    **Duration:** 5 minutes

    **Purpose:** Instant results for repeated queries in same session

    **How it works:**

    ```typescript theme={null}
    // MCP server caches responses locally
    const cache = new Map();

    async function search(query) {
      const cacheKey = `search:${query}`;

      // Check cache first
      if (cache.has(cacheKey)) {
        const cached = cache.get(cacheKey);
        if (Date.now() - cached.timestamp < 5 * 60 * 1000) {
          return cached.results; // 5ms response
        }
      }

      // Cache miss - call API
      const results = await callULPIAPI(query);
      cache.set(cacheKey, { results, timestamp: Date.now() });
      return results; // 200ms response
    }
    ```

    **Benefit:**

    * First query: 200ms (API call)
    * Repeat query: 5ms (local cache)

    **Invalidation:** Automatic after 5 minutes
  </Tab>

  <Tab title="Level 2: MCP Server Process Cache">
    **Location:** Node.js process running MCP server

    **Duration:** 1 hour

    **Purpose:** Persist cache across AI chat restarts

    **Storage:** In-memory (cleared when IDE restarts)

    **Benefit:**

    * Reduces API calls
    * Faster responses throughout work session
    * Survives Claude Desktop window close/reopen (process keeps running)

    **Invalidation:**

    * After 1 hour
    * When you restart IDE completely
  </Tab>

  <Tab title="Level 3: API Cache (Redis)">
    **Location:** ULPI API servers (Redis)

    **Duration:** 5 minutes

    **Purpose:** Fast responses for popular queries across all users

    **How it works:**

    ```php theme={null}
    // Laravel API endpoint
    public function search(Request $request) {
        $cacheKey = "search:{$request->query}:{$request->repository}";

        return Cache::remember($cacheKey, 300, function () use ($request) {
            // Perform vector search in Typesense
            return $this->typesenseSearch($request->query);
        });
    }
    ```

    **Invalidation:**

    * After 5 minutes
    * When repository is re-indexed (webhook)
    * Manual cache clear from dashboard

    **Benefit:**

    * Multiple team members get fast responses
    * Reduces Typesense load
  </Tab>
</Tabs>

### Cache Invalidation on Push

**When you push changes:**

```
git push
↓
Webhook fires
↓
Re-indexing job queued
↓
Files re-indexed
↓
Cache invalidated:
  - API cache (Redis): FLUSH search:*:backend-api
  - MCP server cache: Not invalidated (will expire naturally)
  - Browser cache: Not invalidated (will expire naturally)
```

**Result:** Within 60 seconds, searches return updated documentation

***

## Security & Privacy

**How ULPI protects your private documentation**

<AccordionGroup>
  <Accordion title="🔒 Repository Access: Read-Only" icon="lock">
    **What ULPI can do:**

    * ✅ Read repository contents
    * ✅ Receive webhook notifications
    * ✅ Clone repositories (temporarily, in-memory)

    **What ULPI CANNOT do:**

    * ❌ Write files
    * ❌ Create commits
    * ❌ Push changes
    * ❌ Delete branches
    * ❌ Modify repository settings

    **OAuth tokens:**

    * Stored encrypted (AES-256) in MySQL
    * Never logged or exposed
    * Refreshed automatically before expiration
    * Revocable anytime from GitHub/GitLab settings

    **Revoke access:**

    ```
    GitHub → Settings → Applications → ULPI Documentation → Revoke
    ```

    Access removed instantly.
  </Accordion>

  <Accordion title="🔑 API Keys: Tenant-Scoped" icon="key">
    **Security model:**

    * Each API key is scoped to **your team only**
    * Cannot access other customers' documentation
    * Cannot be used for other ULPI products (unless explicitly granted)

    **Storage:**

    * Hashed using bcrypt (cost factor: 12)
    * Never stored in plaintext
    * Never logged
    * Transmitted only over HTTPS (TLS 1.3)

    **Key format:**

    ```
    ulpi_live_sk_1a2b3c4d5e6f7g8h9i0j...
    ^^^^ ^^^^ ^^
    |    |    |
    |    |    Random 32 characters
    |    Environment (live/test)
    Prefix
    ```

    **Rotation:**

    * Create new key
    * Update MCP config
    * Revoke old key
    * Zero downtime

    **Rate limiting:**

    * 1,000 requests/hour per key
    * Prevents abuse if key is compromised
  </Accordion>

  <Accordion title="📄 Documentation Content: Private" icon="file">
    **What gets indexed:**

    * Documentation files only (`.md`, `.mdx`, `README.*`)
    * NOT source code (unless you explicitly enable comment indexing)

    **AI training policy:**

    <Warning>
      **Your documentation is NEVER used to train AI models.**

      * Not OpenAI's models
      * Not Anthropic's models
      * Not any third-party models

      **What happens:**

      * OpenAI generates embeddings (mathematical representations)
      * OpenAI immediately discards the text (zero data retention policy)
      * Only embeddings are stored (not human-readable)
      * Used exclusively for your search
    </Warning>

    **Data retention:**

    * Indexed as long as repository is connected
    * Deleted within 30 days after repository disconnection
    * Deleted immediately upon account cancellation
  </Accordion>

  <Accordion title="🛡️ Compliance: SOC 2 Type II" icon="shield-check">
    **Certifications:**

    * ✅ SOC 2 Type II compliant
    * ✅ GDPR compliant
    * ✅ CCPA compliant
    * ✅ HIPAA available (Enterprise plan)

    **Data residency:**

    * **US region:** AWS us-east-1 (default)
    * **EU region:** AWS eu-west-1 (Enterprise only)
    * **Custom region:** Available for Enterprise

    **Encryption:**

    * **In transit:** TLS 1.3
    * **At rest:** AES-256 (database, backups, S3)

    **Access controls:**

    * 2FA required for ULPI employees
    * Zero standing access to production
    * Time-limited break-glass access (logged)
    * No access to customer data without explicit approval

    **Auditing:**

    * All access logged to immutable S3 bucket
    * Quarterly security reviews
    * Annual penetration testing

    [Full security details](/security)
  </Accordion>

  <Accordion title="🌐 Network Security" icon="network-wired">
    **Infrastructure:**

    * Hosted on AWS in VPC
    * Private subnets (no public IPs for databases)
    * Security groups (least privilege)
    * WAF (Web Application Firewall) enabled

    **DDoS protection:**

    * CloudFront CDN (global edge caching)
    * AWS Shield Standard
    * Rate limiting at API gateway

    **Monitoring:**

    * Sentry for error tracking
    * CloudWatch for infrastructure
    * Alerts for anomalous traffic
  </Accordion>
</AccordionGroup>

***

## Performance & Scalability

### Response Time Metrics

**Real-world performance data from production:**

| Operation                                   | Average | 95th Percentile | 99th Percentile |
| ------------------------------------------- | ------- | --------------- | --------------- |
| **Simple search** (1-2 keywords)            | 45ms    | 120ms           | 250ms           |
| **Complex search** (full sentence)          | 120ms   | 300ms           | 600ms           |
| **Re-index (small repo)** (under 100 files) | 30s     | 60s             | 90s             |
| **Re-index (medium repo)** (100-1k files)   | 2min    | 5min            | 8min            |
| **Re-index (large repo)** (1k-10k files)    | 5min    | 12min           | 20min           |
| **Webhook processing**                      | 5s      | 15s             | 30s             |

**Why so fast?**

<CardGroup cols={2}>
  <Card title="Vector Search" icon="magnifying-glass" color="#10b981">
    Typesense pre-computes indexes

    **No linear scanning**
  </Card>

  <Card title="Redis Caching" icon="bolt" color="#3b82f6">
    Popular queries cached for 5 minutes

    **Sub-10ms for cached queries**
  </Card>

  <Card title="Chunking Strategy" icon="scissors" color="#8b5cf6">
    Returns 2,000 tokens, not 50,000

    **25x fewer tokens processed**
  </Card>

  <Card title="Dedicated Hardware" icon="server" color="#f59e0b">
    Separate Typesense cluster

    **No database contention**
  </Card>
</CardGroup>

### Scalability Limits

**What ULPI can handle:**

* **Repositories per tenant:** Unlimited (Enterprise)
* **Files per repository:** Unlimited
* **Documentation file size:** Up to 10MB per file
* **Concurrent searches:** 100/second per tenant
* **Total index size:** Average 1MB per 1,000 documentation pages

**Enterprise customers:**

* Dedicated Typesense cluster (isolated resources)
* Custom rate limits
* SLA: 99.9% uptime
* Priority support

***

## Technology Stack

**What powers ULPI Documentation**

<Tabs>
  <Tab title="Backend API">
    **Framework:** Laravel 12.x (PHP 8.2)

    **Why Laravel?**

    * Robust queue system (Horizon)
    * Excellent webhook handling
    * Enterprise-ready
    * Fast development

    **Database:** MySQL 8.0

    * Stores metadata, API keys, user accounts
    * NOT used for search (that's Typesense)

    **Queue:** Redis + Laravel Horizon

    * Background job processing
    * Re-indexing jobs
    * Webhook processing

    **Cache:** Redis

    * Search result caching (5 minutes)
    * Session storage
    * Rate limiting
  </Tab>

  <Tab title="Search & AI">
    **Embeddings:** OpenAI `text-embedding-3-large`

    * 1,536 dimensions
    * Best-in-class semantic understanding
    * Zero data retention policy

    **Vector Search:** Typesense

    * Open-source vector database
    * Sub-50ms similarity search
    * Hybrid semantic + keyword
    * Faceted filtering

    **Similarity Algorithm:** Cosine similarity

    * Range: 0.0 (different) to 1.0 (identical)
    * Threshold: 0.75 for results
  </Tab>

  <Tab title="Infrastructure">
    **Hosting:** AWS (us-east-1)

    * EC2 for API servers
    * RDS for MySQL
    * ElastiCache for Redis
    * S3 for backups

    **CDN:** CloudFront

    * Global edge caching
    * HTTPS termination
    * DDoS protection

    **Monitoring:**

    * Sentry (error tracking)
    * CloudWatch (infrastructure)
    * Uptime Robot (availability)

    **Deployment:**

    * GitHub Actions (CI/CD)
    * Zero-downtime deploys
    * Blue-green deployment strategy
  </Tab>

  <Tab title="MCP Integration">
    **MCP Server:** TypeScript (Node.js)

    **Package:** `@ulpi/mcp-server-documentation`

    **Why TypeScript?**

    * Type safety for API contracts
    * Excellent IDE support
    * Easy to maintain

    **Distribution:** npm (public package)

    **Install:**

    ```bash theme={null}
    npx -y @ulpi/mcp-server-documentation
    ```

    **No installation required** - npx downloads and runs on-demand
  </Tab>
</Tabs>

***

## Comparison: ULPI vs Alternatives

**Why semantic search beats traditional tools**

| Feature               | ULPI                | GitHub Search    | grep/ripgrep       | RAG (DIY)                               |
| --------------------- | ------------------- | ---------------- | ------------------ | --------------------------------------- |
| **Search type**       | Semantic + keyword  | Keyword only     | Keyword only       | Semantic (if configured)                |
| **Natural language**  | ✅ Yes               | ❌ No             | ❌ No               | ✅ Yes (requires setup)                  |
| **Cross-repo search** | ✅ All repos at once | ❌ One at a time  | ❌ Local only       | 🟡 Depends on implementation            |
| **AI integration**    | ✅ 40+ tools via MCP | ❌ No             | ❌ No               | 🟡 Custom integration required          |
| **Auto-sync**         | ✅ Webhook-based     | ✅ Real-time      | ❌ Manual           | 🟡 Must build yourself                  |
| **Setup time**        | ⚡ 5 minutes         | 🟢 0 (built-in)  | 🟢 0 (built-in)    | 🔴 Days/weeks                           |
| **Token efficiency**  | ⚡ 2,000 tokens      | 🔴 50,000 tokens | 🔴 Full file       | 🟡 Depends                              |
| **Latency**           | ⚡ Sub-50ms          | 🟡 100-500ms     | 🟢 Instant (local) | 🔴 Varies                               |
| **Cost**              | 💰 \$29-299/mo      | 🟢 Free          | 🟢 Free            | 🔴 \$\$\$\$ (infrastructure + dev time) |

**When to use each:**

* **ULPI:** AI assistants need semantic understanding across repos
* **GitHub Search:** Finding specific code patterns or file names
* **grep/ripgrep:** Local file searching, exact keywords
* **DIY RAG:** Have ML team, custom requirements, budget >\$50k

***

## Limitations & Edge Cases

**What ULPI doesn't do (yet)**

<Warning>
  **Known limitations:**

  1. **Code search:**
     * ULPI is optimized for documentation, not code
     * Code comments can be indexed (opt-in)
     * Use GitHub/grep for searching actual code

  2. **Binary files:**
     * PDFs, Word docs, images not indexed
     * Convert to Markdown for indexing

  3. **Very large files:**
     * Files >10MB are skipped
     * Break large docs into smaller files

  4. **Real-time (sub-second sync):**
     * Webhook processing takes 30-60 seconds
     * Not suitable for real-time wikis

  5. **Private git servers:**
     * Self-hosted GitHub Enterprise: ✅ Supported
     * GitLab self-hosted: ✅ Supported
     * Other git servers: 🟡 Contact us

  6. **Non-English documentation:**
     * Works, but embeddings optimized for English
     * Other languages: slightly lower accuracy
</Warning>

***

## FAQ: Technical Questions

<AccordionGroup>
  <Accordion title="Can I use a different embedding model?" icon="brain">
    **Not currently.** ULPI uses OpenAI `text-embedding-3-large` for all embeddings.

    **Why?**

    * Best-in-class accuracy
    * 1,536 dimensions (high-fidelity)
    * Proven at scale

    **Enterprise custom models:**

    * Contact us for enterprise plans
    * We can discuss alternative models
    * Requires separate deployment
  </Accordion>

  <Accordion title="How much does re-indexing cost?" icon="dollar-sign">
    **Included in your plan.** No per-search or per-embedding fees.

    **What you pay:**

    * Monthly subscription (\$29-299)
    * Token usage for searches (included in plan)

    **What's free:**

    * Re-indexing (unlimited)
    * Webhook processing
    * Embedding generation
    * Storage

    **Overage:**

    * \$20 per 100,000 additional tokens (search queries only)
  </Accordion>

  <Accordion title="Can I self-host ULPI?" icon="server">
    **Not yet, but coming soon.**

    **Current options:**

    * **Cloud:** Hosted by ULPI (default)
    * **VPC peering:** Connect to your VPC (Enterprise)
    * **On-premise:** Planned for Q2 2025

    **Interested in self-hosting?**

    * Contact [sales@ulpi.io](mailto:sales@ulpi.io)
    * Minimum: Enterprise plan
  </Accordion>

  <Accordion title="What happens if OpenAI's API goes down?" icon="triangle-exclamation">
    **Search continues to work.** Only new indexing is affected.

    **How?**

    * Existing embeddings already in Typesense
    * Search uses those embeddings (no OpenAI API call)
    * Only embedding *generation* requires OpenAI

    **If OpenAI is down:**

    * ✅ Search works normally
    * ❌ New files can't be indexed
    * ❌ Updated files can't be re-indexed

    **Mitigation:**

    * Jobs automatically retry (exponential backoff)
    * Usually resolves in under 15 minutes
  </Accordion>
</AccordionGroup>

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Try It: Getting Started" icon="rocket" href="/documentation/getting-started">
    Set up ULPI in 5 minutes and see semantic search in action

    **No credit card required for trial**
  </Card>

  <Card title="Advanced Search Features" icon="magnifying-glass" href="/documentation/search-features">
    Learn filters, repository scoping, and query optimization

    **Master semantic search**
  </Card>

  <Card title="API Integration" icon="code" href="/documentation/api-integration">
    Integrate ULPI directly into your applications via REST API

    **Build custom workflows**
  </Card>

  <Card title="Repository Management" icon="folder" href="/documentation/repositories">
    Manage connected repos, configure indexing, view metrics

    **Optimize indexing**
  </Card>
</CardGroup>

***

<Note>
  **Still have questions?**

  * 📧 **Email:** [support@ulpi.io](mailto:support@ulpi.io)
  * 📚 **Docs:** [docs.ulpi.io](https://docs.ulpi.io)
  * 💬 **Slack:** [ulpi.io/slack](https://ulpi.io/slack)

  **Average response time:** Under 2 hours during business hours
</Note>