# Document Processing Pipeline

> Understand how documents are processed, chunked, embedded, and indexed for RAG retrieval in your AI voice agent

Source: https://edesy.in/docs/voice-agent/knowledge-base/document-processing

Learn how your documents are transformed from files and web pages into searchable knowledge for your AI voice agent.

> **View status:** [Open your Knowledge Base](https://voice-agent.edesy.in/knowledge-base) to see document processing status and chunk counts.

## Processing Overview

When you upload a document or crawl a URL, it goes through this pipeline:

```
┌─────────────────────────────────────────────────────────────────┐
│                    Document Processing Pipeline                  │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Input                                                         │
│   ├── PDF/DOCX Upload ─────┐                                    │
│   └── URL Crawl ───────────┘                                    │
│              │                                                  │
│              ▼                                                  │
│   ┌─────────────────┐                                           │
│   │    Parsing      │  Extract text from source                 │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │    Chunking     │  Split into ~500 token pieces             │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │   Embedding     │  Convert to vectors (OpenAI)              │
│   └────────┬────────┘                                           │
│            │                                                    │
│            ▼                                                    │
│   ┌─────────────────┐                                           │
│   │    Indexing     │  Store in Pinecone                        │
│   └─────────────────┘                                           │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

## Pipeline Stages

### 1. Parsing

**Purpose:** Extract raw text from the source.

| Source | Parser | Output |
|--------|--------|--------|
| PDF | PDF.js / pdfparse | Plain text |
| DOCX | mammoth | Plain text + structure |
| TXT/MD | Direct read | Plain text |
| HTML (URL) | Cheerio | Cleaned text |

**What's extracted:**
- Main content text
- Headings and structure
- Lists and tables (as text)

**What's ignored:**
- Images (no OCR currently)
- JavaScript/CSS
- Navigation elements
- Headers/footers

### 2. Chunking

**Purpose:** Split text into smaller, retrievable pieces.

| Parameter | Value | Description |
|-----------|-------|-------------|
| **Chunk Size** | ~500 tokens | Approximate size per chunk |
| **Overlap** | ~50 tokens | Context preserved between chunks |
| **Boundary** | Sentences | Chunks end at sentence boundaries |

**Why chunk?**

1. **LLM context limits**: Can't send entire documents to LLM
2. **Precision**: Smaller chunks = more targeted retrieval
3. **Relevance**: Retrieve only relevant sections

**Example chunking:**

```
Original document (1500 words):
┌─────────────────────────────────────────────────┐
│ Section 1: Introduction (300 words)             │
│ Section 2: Features (500 words)                 │
│ Section 3: Pricing (400 words)                  │
│ Section 4: FAQ (300 words)                      │
└─────────────────────────────────────────────────┘

After chunking (4 chunks):
┌────────────────┐  ┌────────────────┐
│ Chunk 1        │  │ Chunk 2        │
│ Intro + start  │  │ Features       │
│ of Features    │  │ continued      │
└────────────────┘  └────────────────┘

┌────────────────┐  ┌────────────────┐
│ Chunk 3        │  │ Chunk 4        │
│ Pricing        │  │ FAQ            │
└────────────────┘  └────────────────┘
```

### 3. Embedding

**Purpose:** Convert text chunks to vectors for semantic search.

| Setting | Value |
|---------|-------|
| **Model** | OpenAI text-embedding-3-small |
| **Dimensions** | 1536 |
| **Batch Size** | 100 chunks |

**How it works:**

```
Text: "Our return policy allows returns within 30 days"
        │
        ▼
   OpenAI Embedding API
        │
        ▼
Vector: [0.023, -0.156, 0.089, ..., 0.045]  (1536 dimensions)
```

**Why embeddings?**

- Similar meaning = similar vectors
- Enables semantic search (understanding, not just keywords)
- "return policy" matches "refund rules"

### 4. Indexing

**Purpose:** Store vectors for fast similarity search.

| Setting | Value |
|---------|-------|
| **Database** | Pinecone |
| **Namespace** | Per knowledge base |
| **Metadata** | Document ID, text, source |

**Stored metadata per chunk:**

```json
{
  "documentId": "doc_abc123",
  "documentName": "Return Policy",
  "knowledgeBaseId": "kb_xyz789",
  "chunkIndex": 0,
  "text": "Our return policy allows...",
  "wordCount": 150,
  "sourceType": "file",
  "sourceUrl": null
}
```

## Document Status

Each document displays a status badge:

| Status | Icon | Description | Duration |
|--------|------|-------------|----------|
| **Pending** | Clock | Queued for processing | Seconds |
| **Processing** | Spinner | Pipeline in progress | Seconds to minutes |
| **Completed** | Checkmark | Ready for retrieval | - |
| **Failed** | X | Error occurred | - |
| **Stuck** | Warning | Processing timed out (>10 min) | - |

### Status Flow

```
PENDING ──► PROCESSING ──► COMPLETED
                │
                ├──► FAILED (error)
                │
                └──► STUCK (timeout)
```

### Handling Failed Documents

When a document fails:

1. **View error**: Click the info icon to see the error message
2. **Fix issue**: Address the cause (file format, content, etc.)
3. **Retry**: Click the "Retry" button

Common failures:
- Invalid file format
- Empty content
- Embedding API timeout
- Network issues

See [Troubleshooting](https://edesy.in/docs/voice-agent/knowledge-base/troubleshooting) for solutions.

### Handling Stuck Documents

Documents stuck in "Processing" for over 10 minutes are marked as "Stuck":

1. **Identify**: Look for yellow "Stuck" badge
2. **Retry**: Click the "Retry" button
3. **Monitor**: Watch for successful completion

**Why documents get stuck:**
- Server restart during processing
- Embedding API timeout
- Database connection lost

## Retrieval Process

When your agent needs information during a call:

```
User: "What's your return policy?"
              │
              ▼
        ┌───────────┐
        │  Embed    │  Convert question to vector
        │  Query    │
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │  Vector   │  Find similar chunks in Pinecone
        │  Search   │
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │  Rank &   │  Apply minScore filter
        │  Filter   │  Return top K results
        └─────┬─────┘
              │
              ▼
        ┌───────────┐
        │   LLM     │  Generate response using
        │ Response  │  retrieved context
        └───────────┘
```

**Configuration options:**

| Setting | Default | Description |
|---------|---------|-------------|
| `topK` | 5 | Number of chunks to retrieve |
| `minScore` | 0.7 | Minimum relevance (0-1) |

Learn more: [RAG Integration](https://edesy.in/docs/voice-agent/knowledge-base/rag-integration)

## Processing Time

Typical processing times:

| Document Type | Size | Processing Time |
|---------------|------|-----------------|
| Small PDF | <1 MB | 5-15 seconds |
| Large PDF | 1-10 MB | 30-90 seconds |
| Single URL | - | 10-30 seconds |
| Sitemap (10 pages) | - | 2-5 minutes |
| Sitemap (50 pages) | - | 10-15 minutes |

**Factors affecting time:**
- File size and complexity
- Number of pages/chunks
- Current system load
- Embedding API response time

## Monitoring Processing

### Document List View

In your knowledge base, see:
- Document name and source
- Processing status
- Chunk count (when completed)
- Error messages (when failed)

### Chunk Count

The chunk count indicates how many pieces your document was split into:

| Chunk Count | Document Size |
|-------------|---------------|
| 1-5 | Small document |
| 5-20 | Medium document |
| 20-50 | Large document |
| 50+ | Very large document |

### Error Details

For failed documents:
1. Click the info icon
2. View full error message
3. See processing metadata
4. Check source URL (for crawled content)

## Best Practices

### For Better Chunking

- Use clear headings (H1, H2, H3)
- Write in paragraphs, not walls of text
- Keep related content together
- Avoid very long sentences

### For Better Retrieval

- One topic per document
- Use specific, descriptive titles
- Include common question phrases
- Avoid abbreviations without definitions

### For Monitoring

- Check processing status after uploads
- Retry failed documents promptly
- Review chunk counts for expected values
- Test retrieval with sample questions

## Next Steps

- [RAG Integration](https://edesy.in/docs/voice-agent/knowledge-base/rag-integration) - Configure retrieval settings
- [Troubleshooting](https://edesy.in/docs/voice-agent/knowledge-base/troubleshooting) - Fix processing issues
- [Upload Documents](https://edesy.in/docs/voice-agent/knowledge-base/uploading-documents) - Import files
- [Web Crawling](https://edesy.in/docs/voice-agent/knowledge-base/web-crawling) - Import web content

---

**Questions about processing?** Check [Troubleshooting](https://edesy.in/docs/voice-agent/knowledge-base/troubleshooting) or contact support.