Free tool
AI bot access test
Enter an address to see which AI bots your robots.txt lets in, which it blocks and what that means.
This test reads your site’s robots.txt and shows whether 14 bots may enter your page. It groups bots by role: search and answers, user requests and model training. Blocking search bots makes it harder to be cited in that engine’s answers; blocking training bots is a business decision.
robots.txt addition that closes training and keeps search open
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: CCBot Disallow: /
Once you write a group for a specific bot, that bot no longer reads the rules in the User-agent: * group (RFC 9309). If you have paths to keep private, add them to these groups too.
Why are bots split into three groups?
| Group | Examples | If blocked |
|---|---|---|
| Search and answers | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot, Meta-WebIndexer | The site may not be cited in that engine’s answers; for example, sites that opt out of OAI-SearchBot are not shown in ChatGPT search answersSource 1: Overview of OpenAI crawlers. |
| User requests | ChatGPT-User, Claude-User, Perplexity-User | Some may not follow robots.txt: OpenAI says rules may not apply to ChatGPT-UserSource 1: Overview of OpenAI crawlers, and Perplexity says Perplexity-User generally ignores themSource 3: Perplexity crawlers. Claude-User follows themSource 2: Does Anthropic crawl data from the web, and how can site owners block the crawler?. |
| Model training | GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, CCBot | Signals that content should not be used in future model trainingSource 1: Overview of OpenAI crawlersSource 2: Does Anthropic crawl data from the web, and how can site owners block the crawler?. Google-Extended does not affect inclusion or ranking in Google SearchSource 4: Google’s common crawlers (Googlebot, Google-Extended). |
What should you watch for when adding robots.txt rules?
Under RFC 9309, the robots.txt standard, a bot that finds a group matching its name follows that group; if none matches, it follows the User-agent: * groupSource 6: RFC 9309: Robots Exclusion Protocol (Eylül 2022). So once you write a group for a specific bot, that bot no longer reads the rules in the * group. Paths you want to keep private must be added to the bot-specific groups too.
The test only reads robots.txt. If a firewall or bot protection blocks a bot, we cannot see it; check your server logs.
Frequently asked questions
01Should I block training bots?
It is a business decision. Blocking training bots does not directly affect search visibility; because search and training bots are defined separately, you can decide for eachSource 1: Overview of OpenAI crawlersSource 2: Does Anthropic crawl data from the web, and how can site owners block the crawler?.
02Which Meta bot matters for Meta AI?
The bot related to Meta AI’s search results is Meta-WebIndexer; Meta-ExternalAgent is for model training and product improvementSource 5: Meta Web Crawlers.
03Why are Grok and DeepSeek not listed?
Neither company publishes an official crawler name that site owners can target in robots.txt. We do not add unverified names to the test; details on the engine pages.
Sources
- [1]Overview of OpenAI crawlersOpenAI · accessed: 26 September 2026
- [2]Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropic · accessed: 26 September 2026
- [3]Perplexity crawlersPerplexity · accessed: 26 September 2026
- [4]Google’s common crawlers (Googlebot, Google-Extended)Google Search Central · accessed: 26 September 2026
- [5]Meta Web CrawlersMeta for Developers · accessed: 26 September 2026
- [6]RFC 9309: Robots Exclusion Protocol (Eylül 2022)IETF / RFC Editor · accessed: 26 September 2026
How to cite this page
Kaynaq. (26 September 2026). AI bot access test. https://kaynaq.pages.dev/en/tools/ai-bot-access-test
Updates and corrections
No corrections on this page yet. If you spot an error, write to us; we publish every fix here with its date.