Document Processing Pipeline

Learn how your documents are transformed from files and web pages into searchable knowledge for your AI voice agent.

View status: Open your Knowledge Base to see document processing status and chunk counts.

Processing Overview

When you upload a document or crawl a URL, it goes through this pipeline:

┌─────────────────────────────────────────────────────────────────┐
│                    Document Processing Pipeline                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Input                                                         │
│   ├── PDF/DOCX Upload ─────┐                                    │
│   └── URL Crawl ───────────┘                                    │
│              │                                                  │
│              ▼                                                  │
│   ┌─────────────────┐                                           │
│   │    Parsing      │  Extract text from source                 │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │    Chunking     │  Split into ~500 token pieces             │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │   Embedding     │  Convert to vectors (OpenAI)              │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │    Indexing     │  Store in Pinecone                        │
│   └─────────────────┘                                           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Pipeline Stages

1. Parsing

Purpose: Extract raw text from the source.

What's extracted:

• Main content text
• Headings and structure
• Lists and tables (as text)

What's ignored:

• Images (no OCR currently)
• JavaScript/CSS
• Navigation elements
• Headers/footers

2. Chunking

Purpose: Split text into smaller, retrievable pieces.

Why chunk?

1. LLM context limits: Can't send entire documents to LLM
2. Precision: Smaller chunks = more targeted retrieval
3. Relevance: Retrieve only relevant sections

Example chunking:

Original document (1500 words):
┌─────────────────────────────────────────────────┐
│ Section 1: Introduction (300 words)             │
│ Section 2: Features (500 words)                 │
│ Section 3: Pricing (400 words)                  │
│ Section 4: FAQ (300 words)                      │
└─────────────────────────────────────────────────┘

After chunking (4 chunks):
┌────────────────┐  ┌────────────────┐
│ Chunk 1        │  │ Chunk 2        │
│ Intro + start  │  │ Features       │
│ of Features    │  │ continued      │
└────────────────┘  └────────────────┘

┌────────────────┐  ┌────────────────┐
│ Chunk 3        │  │ Chunk 4        │
│ Pricing        │  │ FAQ            │
└────────────────┘  └────────────────┘

3. Embedding

Purpose: Convert text chunks to vectors for semantic search.

How it works:

Text: "Our return policy allows returns within 30 days"
        │
        ▼
   OpenAI Embedding API
        │
        ▼
Vector: [0.023, -0.156, 0.089, ..., 0.045]  (1536 dimensions)

Why embeddings?

• Similar meaning = similar vectors
• Enables semantic search (understanding, not just keywords)
• "return policy" matches "refund rules"

4. Indexing

Purpose: Store vectors for fast similarity search.

Stored metadata per chunk:

{
  "documentId": "doc_abc123",
  "documentName": "Return Policy",
  "knowledgeBaseId": "kb_xyz789",
  "chunkIndex": 0,
  "text": "Our return policy allows...",
  "wordCount": 150,
  "sourceType": "file",
  "sourceUrl": null
}

Document Status

Each document displays a status badge:

Status Flow

PENDING ──► PROCESSING ──► COMPLETED
                │
                ├──► FAILED (error)
                │
                └──► STUCK (timeout)

Handling Failed Documents

When a document fails:

1. View error: Click the info icon to see the error message
2. Fix issue: Address the cause (file format, content, etc.)
3. Retry: Click the "Retry" button

Common failures:

• Invalid file format
• Empty content
• Embedding API timeout
• Network issues

See Troubleshooting for solutions.

Handling Stuck Documents

Documents stuck in "Processing" for over 10 minutes are marked as "Stuck":

1. Identify: Look for yellow "Stuck" badge
2. Retry: Click the "Retry" button
3. Monitor: Watch for successful completion

Why documents get stuck:

• Server restart during processing
• Embedding API timeout
• Database connection lost

Retrieval Process

When your agent needs information during a call:

User: "What's your return policy?"
              │
              ▼
        ┌───────────┐
        │  Embed    │  Convert question to vector
        │  Query    │
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │  Vector   │  Find similar chunks in Pinecone
        │  Search   │
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │  Rank &   │  Apply minScore filter
        │  Filter   │  Return top K results
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │   LLM     │  Generate response using
        │ Response  │  retrieved context
        └───────────┘

Configuration options:

Learn more: RAG Integration

Processing Time

Typical processing times:

Factors affecting time:

• File size and complexity
• Number of pages/chunks
• Current system load
• Embedding API response time

Monitoring Processing

Document List View

In your knowledge base, see:

• Document name and source
• Processing status
• Chunk count (when completed)
• Error messages (when failed)

Chunk Count

The chunk count indicates how many pieces your document was split into:

Error Details

For failed documents:

1. Click the info icon
2. View full error message
3. See processing metadata
4. Check source URL (for crawled content)

Best Practices

For Better Chunking

• Use clear headings (H1, H2, H3)
• Write in paragraphs, not walls of text
• Keep related content together
• Avoid very long sentences

For Better Retrieval

• One topic per document
• Use specific, descriptive titles
• Include common question phrases
• Avoid abbreviations without definitions

For Monitoring

• Check processing status after uploads
• Retry failed documents promptly
• Review chunk counts for expected values
• Test retrieval with sample questions

Next Steps

• RAG Integration - Configure retrieval settings
• Troubleshooting - Fix processing issues
• Upload Documents - Import files
• Web Crawling - Import web content

Questions about processing? Check Troubleshooting or contact support.