Anthropic · training bot · Last updated 2026-05-22
ClaudeBot robots.txt — Allow, Block, or Audit
How to allow or block ClaudeBot and the related Anthropic crawlers (Claude-SearchBot, Claude-User, anthropic-ai) in your robots.txt. Required directives + recommended setup.
The bot
What does ClaudeBot do?
ClaudeBot is Anthropic's primary training-data crawler for Claude. Anthropic operates four related user-agents: ClaudeBot (training), Claude-SearchBot (real-time search indexing for Claude's web-search feature), Claude-User (on-demand fetches when a Claude user references a specific URL), and the legacy anthropic-ai user-agent (older training crawler still respected). Each is a separate User-agent block in robots.txt.
Why it matters
Why ClaudeBot matters for AI visibility
Claude is a knowledge-presence engine — it answers from training corpus, not live retrieval (for most queries). Allowing ClaudeBot is how your content enters that training corpus and becomes citable when users ask Claude questions about your brand or category. Blocking ClaudeBot caps your Claude visibility growth. Anthropic publishes its bot allowlist transparently at docs.anthropic.com — verify the user-agent strings against the canonical source.
Copy and paste
Copy-paste robots.txt block
The exact directive to add to your robots.txt for ClaudeBot. Paste at the end of your file — bot-specific blocks override the wildcard above.
# Allow all four Anthropic / Claude crawlers
User-agent: ClaudeBot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
User-agent: Claude-User
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboard
# Legacy user-agent — still in some training pipelines
User-agent: anthropic-ai
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /dashboardFAQ
Frequently asked questions about ClaudeBot
What's the difference between ClaudeBot and Claude-User?
ClaudeBot crawls broadly for training data — Anthropic schedules its visits. Claude-User performs on-demand fetches when a Claude user pastes or references a URL in conversation. Blocking ClaudeBot opts you out of training; blocking Claude-User makes Claude return "I couldn't fetch that URL" to users who reference your page directly. Almost always allow both.
Is anthropic-ai the same as ClaudeBot?
anthropic-ai is the legacy user-agent from before Anthropic rebranded its training crawler to ClaudeBot. Older Anthropic training pipelines may still send it; some robots.txt files written 2023-2024 only have anthropic-ai listed. Best practice: include both ClaudeBot and anthropic-ai blocks for compatibility.
Where does Anthropic document its bots?
https://docs.anthropic.com/en/docs/build-with-claude/web-crawling is the canonical source. It lists all four current user-agents (ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai), explains what each does, and confirms they respect standard robots.txt rules.
How fast does ClaudeBot crawl?
Anthropic doesn't publish exact rate limits but it's significantly slower than Googlebot. A medium-sized site (5,000-10,000 pages) typically sees ClaudeBot complete a full crawl over several weeks. Don't set Crawl-delay aggressively; the natural pace is already conservative.
Will blocking ClaudeBot remove existing Claude knowledge of my brand?
No, not immediately. Content already in the training corpus persists in current Claude models. Blocking ClaudeBot prevents future inclusion, so the impact compounds over the next training cycle (months to a year). For immediate removal requests, Anthropic provides a separate opt-out form for specific content.
Keep going
Related AI crawlers
Each major AI engine operates one or more user-agents. Configure them in parallel for complete coverage.
Written and maintained by the BrandCited Editorial Team. Last updated .
Free AI visibility check
Want a full AI visibility check?
Enter your website. See what AI assistants say about your business, who they name instead and what to fix.
The free check asks
- ChatGPT
- Claude
- Gemini
- Perplexity
Your report shows
- What AI says about youWhether ChatGPT, Claude, Gemini and Perplexity know your business
- Who AI names insteadThe businesses suggested in your category
- What to fix on your siteAI crawler access, structured data and llms.txt, top fixes first

