Web Crawling for Knowledge Base

Import content directly from websites into your knowledge base. Crawl single pages, multiple URLs, or entire websites using sitemaps.

Start crawling: Open your Knowledge Base, select a knowledge base, and click Add Web Content.

Looking to upload files instead? See Upload Documents.

Why Crawl Websites?

Ideal for:

• FAQ pages that update frequently
• Product documentation websites
• Help center articles
• Blog posts and guides
• Pricing pages

Crawl Methods

Single Page

Import one specific page:

https://example.com/faq

Use when:

• You need a specific page
• Testing before full crawl
• Content is on one page

Sitemap Crawl

Import multiple pages from a sitemap:

https://example.com/sitemap.xml

Use when:

• Importing entire website sections
• Bulk importing many pages
• Content is well-organized

Step-by-Step Guide

1. Open Add Web Content

1. Go to Knowledge Base in the sidebar
2. Click on your knowledge base
3. Find the Add Web Content tab (or expand the upload section)

2. Enter URL

Enter a URL or sitemap URL:

3. Configure Options (Sitemap)

For sitemap crawls, configure:

4. Preview & Select Pages

For sitemaps, you'll see a list of discovered URLs:

✓ /faq                    FAQ Page
✓ /pricing                Pricing
✓ /features               Features
□ /blog/post-1            Blog Post 1
□ /blog/post-2            Blog Post 2

Select which pages to import (all are selected by default, up to max pages).

5. Start Crawling

Click Start Crawling to begin:

1. Crawling: Pages are fetched and content extracted
2. Processing: Text is chunked and embedded
3. Indexing: Vectors stored in Pinecone

6. Monitor Progress

Watch the crawl progress:

Crawling: 15/50 pages
├── ✓ Completed: 12
├── ⏳ Processing: 2
└── ✗ Failed: 1

Content Extraction

The crawler extracts:

Extraction Quality

Best extraction:

• Clean HTML with semantic markup
• Blog posts and articles
• Documentation pages
• FAQ pages

Challenging extraction:

• Single-page apps (JavaScript rendered)
• Pages with mostly images
• Login-protected content
• Dynamically loaded content

Sitemap Requirements

Supported Formats

Example Sitemap

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/faq</loc>
    <lastmod>2024-01-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2024-01-10</lastmod>
  </url>
</urlset>

Finding Your Sitemap

Common locations:

• https://yoursite.com/sitemap.xml
• https://yoursite.com/sitemap_index.xml
• Check robots.txt for sitemap location

Document Status

Each crawled page shows a status:

Retry Failed Documents

If a crawl fails:

1. Click the Retry button next to the failed document
2. The system will re-crawl the URL
3. Re-process and re-index the content

Common failure reasons:

• Temporary network issues
• Rate limiting by the website
• Page structure changed
• Embedding API timeout

See Troubleshooting for more details.

Re-Crawling for Updates

To update crawled content:

1. Delete the existing URL documents
2. Re-crawl the same URLs

Tip: Keep track of which URLs you've crawled. The document list shows the source URL for each entry.

Best Practices

Start Small

1. Crawl a single page first
2. Verify content extraction quality
3. Then crawl more pages

Respect Rate Limits

• Use the default 2-second delay
• Don't crawl too many pages at once
• Respect robots.txt rules

Select Relevant Pages

Don't import everything. Focus on:

• FAQ and support pages
• Product documentation
• Policy pages
• Key landing pages

Avoid:

• Blog archives (hundreds of old posts)
• Image galleries
• Login/account pages
• Terms and privacy (unless needed)

Monitor Quality

After crawling:

1. Check document status for failures
2. Test RAG retrieval in the agent
3. Remove low-quality extractions

Limits & Quotas

Troubleshooting

"Failed to crawl URL"

Possible causes:

• Website blocks crawlers (check robots.txt)
• Page requires authentication
• Invalid URL format
• Network timeout

Solutions:

• Verify URL is accessible in browser
• Check if site allows crawling
• Try single page instead of sitemap

"No content extracted"

Possible causes:

• JavaScript-rendered content
• Page has no text content
• Content behind login

Solutions:

• Try a different page
• Use file upload for this content
• Check page loads without JavaScript

Stuck documents

Documents stuck in "Processing" for more than 10 minutes can be retried:

1. Look for the yellow "Stuck" badge
2. Click Retry to re-process

See Troubleshooting for more solutions.

Use Cases

Import Company FAQ

Sitemap: https://company.com/sitemap.xml
Selected pages:
- /faq
- /help/shipping
- /help/returns
- /help/payment

Import Product Docs

Sitemap: https://docs.company.com/sitemap.xml
Max pages: 50
Selected: All pages in /product-guide/

Import Knowledge Center

Single URL: https://support.company.com/knowledge-base
Then selectively crawl individual articles

Next Steps

• Document Processing - How content is indexed
• RAG Integration - Configure retrieval
• Troubleshooting - Fix crawl issues
• Upload Documents - Import local files

Ready to import web content? Open the Edesy Platform to start crawling.