Engines · Claude
Visibility in Claude: Anthropic’s three bots and robots.txt decisions
Claude can search the web and cite sources when a user asks. Anthropic uses three separate bots for this and states in its own documentation what blocking each one does. This page explains those differences and how visibility in Claude is measured.
Anthropic has three bots: ClaudeBot collects content for training, Claude-SearchBot indexes content to improve search result quality, and Claude-User visits pages when a user asks a question. All three respect robots.txt. Blocking ClaudeBot stops training use; blocking Claude-SearchBot or Claude-User can reduce your visibility in Claude’s search answers.
Which Anthropic bots visit your site?
According to Anthropic, its bots honour standard robots.txt “do not crawl” directives and also support Crawl-delay, a non-standard extensionSource 1: Does Anthropic crawl data from the web, and how can site owners block the crawler?. To close training and stay visible in search, block ClaudeBot and allow Claude-SearchBot and Claude-User.
How do we measure Claude?
- The question set is asked the same way every month in answers where Claude uses web search; each question twice.
- The answer text, mentions, source list and accuracy are recorded with the date.
- Anthropic’s documentation does not detail source selection, so we see which sites are cited by measuring.
Frequently asked questions
01If I block ClaudeBot, do I disappear from Claude?
ClaudeBot is for training; search visibility relates to Claude-SearchBot and Claude-UserSource 1: Does Anthropic crawl data from the web, and how can site owners block the crawler?. You can block ClaudeBot and allow the other two.
02Can I limit how fast the bots crawl?
Yes. Anthropic says its bots support the Crawl-delay directive in robots.txtSource 1: Does Anthropic crawl data from the web, and how can site owners block the crawler?.
03Is my site open to Claude’s bots?
For robots.txt, our readiness test shows all three bots separately. Firewall rules have to be checked in your server logs.
Sources
- [1]Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropic · accessed: 26 September 2026
How to cite this page
Kaynaq. (26 September 2026). Visibility in Claude: Anthropic’s three bots and robots.txt decisions. https://kaynaq.pages.dev/en/ai-engines/claude
Updates and corrections
No corrections on this page yet. If you spot an error, write to us; we publish every fix here with its date.