Create robots.txt files for your website. Control how search engines and AI crawlers access your site with Allow, Disallow rules, and sitemap references.
Place robots.txt in your website root directory (e.g., example.com/robots.txt)
| Directive | Description | Example |
|---|---|---|
| User-agent | Specifies which crawler the rules apply to | User-agent: Googlebot |
| Disallow | Blocks access to a path or pattern | Disallow: /admin/ |
| Allow | Explicitly permits access to a path | Allow: /api/public |
| Sitemap | Points to XML sitemap location | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Seconds between requests | Crawl-delay: 10 |
| Host | Preferred domain (Yandex) | Host: https://www.example.com |
User-agent: * Disallow: /cart Disallow: /checkout Disallow: /account Disallow: /admin/ Disallow: /api/ Disallow: /*?* User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / Sitemap: https://example.com/sitemap.xml
Visual Rule Builder
Multiple User Agents
Quick Templates
Sitemap Integration
AI Bot Blocking
Advanced Options
Control which pages search engines can index. Block admin areas, duplicate content, and internal pages.
Prevent AI companies from using your content for training by blocking GPTBot, CCBot, and similar crawlers.
Block cart, checkout, and account pages from indexing while allowing product pages to be crawled.
Block all crawlers from staging or development environments to prevent them from appearing in search results.
Love our free tools? Discover our AI-powered products that help businesses automate customer communication.
Automate phone calls with AI. Handle appointments, inquiries, and support 24/7 in 40+ languages.
Build WhatsApp chatbots without coding. Automate support and boost sales with 98% open rates.
Add an intelligent chatbot to your website. Convert visitors, answer questions, and capture leads 24/7.
Translate and dub videos in 100+ languages with natural AI voices. Reach global audiences in minutes.
Turn every call into insights with AI transcription, sentiment analysis, and quality scoring for support and sales teams.
Visual deal pipeline with drag-and-drop kanban, Gmail integration, and contact management. Close more deals faster.
Connect customers and staff on a masked number so personal numbers stay private. Click-to-call masking with a simple REST API.
Local, toll-free, and virtual numbers across India from Rs 349/month. Call forwarding, IVR, and free number porting included.
Elastic SIP trunking at 50 paisa/min. Connect any PBX or AI voice agent to the Indian PSTN. No contracts, self-serve setup.
Recurring revenue for Shopify. Pause/skip, build-a-box, prepaid plans, and a customer portal with UPI Autopay & Razorpay.