Every check traces back to a source

Claude-User handles on-demand URL fetches from Claude.ai users.

• claude_user_ondemand — User-initiated fetch only

ClaudeBot is Anthropic combined training + retrieval crawler.

• claudebot_combined — Single bot for both training and retrieval

Applebot-Extended controls Apple Intelligence training use. Separate from Applebot indexing.

Bytespider is ByteDance training-focused crawler. Known for aggressive rates.

Google-Extended is a robots.txt token (not a crawler) that controls Gemini + Bard training use. Separate from Googlebot.

• google_extended_training_opt_out — Robots.txt token, not a crawler
• independent_from_googlebot — Googlebot indexing unaffected by Google-Extended rules

ChatGPT-User is on-demand, not a crawler. Blocking it hurts user-initiated citations.

• chatgpt_user_ondemand — Fetches only URLs user references

OpenAI GPTBot crawler for training data. Respects robots.txt. Not used for real-time retrieval.

• gptbot_user_agent — User-agent token is GPTBot
• gptbot_respects_robots — Honors robots.txt Disallow rules

Perplexity-User is the on-demand URL fetch agent for Perplexity sessions.

PerplexityBot indexes content for Perplexity citation-heavy answers.

• perplexity_citations — Perplexity returns linked citations for cited sources