Web Crawling for Knowledge Base

Import content directly from websites into your knowledge base. Crawl single pages, multiple URLs, or entire websites using sitemaps.

Start crawling: Open your Knowledge Base, select a knowledge base, and click Add Web Content.

Looking to upload files instead? See Upload Documents.

Why Crawl Websites?#

Ideal for:

• FAQ pages that update frequently
• Product documentation websites
• Help center articles
• Blog posts and guides
• Pricing pages

Crawl Methods#

Single Page#

Import one specific page:

https://example.com/faq

Use when:

• You need a specific page
• Testing before full crawl
• Content is on one page

Sitemap Crawl#

Import multiple pages from a sitemap:

https://example.com/sitemap.xml

Use when:

• Importing entire website sections
• Bulk importing many pages
• Content is well-organized

Step-by-Step Guide#

1. Open Add Web Content#

1. Go to Knowledge Base in the sidebar
2. Click on your knowledge base
3. Find the Add Web Content tab (or expand the upload section)

2. Enter URL#

Enter a URL or sitemap URL:

3. Configure Options (Sitemap)#

For sitemap crawls, configure:

4. Preview & Select Pages#

For sitemaps, you'll see a list of discovered URLs:

✓ /faq                    FAQ Page
✓ /pricing                Pricing
✓ /features               Features
□ /blog/post-1            Blog Post 1
□ /blog/post-2            Blog Post 2

Select which pages to import (all are selected by default, up to max pages).

5. Start Crawling#

Click Start Crawling to begin:

1. Crawling: Pages are fetched and content extracted
2. Processing: Text is chunked and embedded
3. Indexing: Vectors stored in Pinecone

6. Monitor Progress#

Watch the crawl progress:

Crawling: 15/50 pages
├── ✓ Completed: 12
├── ⏳ Processing: 2
└── ✗ Failed: 1

Content Extraction#

The crawler extracts:

Extraction Quality#

Best extraction:

• Clean HTML with semantic markup
• Blog posts and articles
• Documentation pages
• FAQ pages

Challenging extraction:

• Single-page apps (JavaScript rendered)
• Pages with mostly images
• Login-protected content
• Dynamically loaded content

Sitemap Requirements#

Supported Formats#

Example Sitemap#

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/faq</loc>
    <lastmod>2024-01-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2024-01-10</lastmod>
  </url>
</urlset>

Finding Your Sitemap#

Common locations:

• https://yoursite.com/sitemap.xml
• https://yoursite.com/sitemap_index.xml
• Check robots.txt for sitemap location

Document Status#

Each crawled page shows a status:

Retry Failed Documents#

If a crawl fails:

1. Click the Retry button next to the failed document
2. The system will re-crawl the URL
3. Re-process and re-index the content

Common failure reasons:

• Temporary network issues
• Rate limiting by the website
• Page structure changed
• Embedding API timeout

See Troubleshooting for more details.

Re-Crawling for Updates#

To update crawled content:

1. Delete the existing URL documents
2. Re-crawl the same URLs

Best Practices#

Start Small#

1. Crawl a single page first
2. Verify content extraction quality
3. Then crawl more pages

Respect Rate Limits#

• Use the default 2-second delay
• Don't crawl too many pages at once
• Respect robots.txt rules

Select Relevant Pages#

Don't import everything. Focus on:

• FAQ and support pages
• Product documentation
• Policy pages
• Key landing pages

Avoid:

• Blog archives (hundreds of old posts)
• Image galleries
• Login/account pages
• Terms and privacy (unless needed)

Monitor Quality#

After crawling:

1. Check document status for failures
2. Test RAG retrieval in the agent
3. Remove low-quality extractions

Limits & Quotas#

Troubleshooting#

"Failed to crawl URL"#

Possible causes:

• Website blocks crawlers (check robots.txt)
• Page requires authentication
• Invalid URL format
• Network timeout

Solutions:

• Verify URL is accessible in browser
• Check if site allows crawling
• Try single page instead of sitemap

"No content extracted"#

Possible causes:

• JavaScript-rendered content
• Page has no text content
• Content behind login

Solutions:

• Try a different page
• Use file upload for this content
• Check page loads without JavaScript

Stuck documents#

Documents stuck in "Processing" for more than 10 minutes can be retried:

1. Look for the yellow "Stuck" badge
2. Click Retry to re-process

See Troubleshooting for more solutions.

Use Cases#

Import Company FAQ#

Sitemap: https://company.com/sitemap.xml
Selected pages:
- /faq
- /help/shipping
- /help/returns
- /help/payment

Import Product Docs#

Sitemap: https://docs.company.com/sitemap.xml
Max pages: 50
Selected: All pages in /product-guide/

Import Knowledge Center#

Single URL: https://support.company.com/knowledge-base
Then selectively crawl individual articles

Next Steps#

• Document Processing - How content is indexed
• RAG Integration - Configure retrieval
• Troubleshooting - Fix crawl issues
• Upload Documents - Import local files

Ready to import web content? Open the Edesy Platform to start crawling.