AI crawlers and technical access: robots.txt, JavaScript and llms.txt

1. Which bots are there and what do they do?

AI companies run separate bots for search and answers, model training and user requests. Examples from the companies’ own documentation:

Details per engine are on the AI engines pages.

2. Write robots.txt correctly

• Do not block search and answer bots; being cited in that engine’s answers becomes harder. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers^{[1]Source 1: Overview of OpenAI crawlersOverview of OpenAI crawlersOpenAIaccessed: 26 September 2026}.
• Decide separately for training bots. Because search and training bots are defined separately, you can close training and keep search open^{[1]Source 1: Overview of OpenAI crawlersOverview of OpenAI crawlersOpenAIaccessed: 26 September 2026[2]Source 2: Does Anthropic crawl data from the web, and how can site owners block the crawler?Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropicaccessed: 26 September 2026}.
• Once you write a group for a specific bot, it no longer reads the User-agent: * group; under RFC 9309 the * group applies only when no group matches^{[6]Source 6: RFC 9309: Robots Exclusion Protocol (Eylül 2022)RFC 9309: Robots Exclusion Protocol (Eylül 2022)IETF / RFC Editoraccessed: 26 September 2026}. Add private paths to bot-specific groups too.
• Some user-request bots may not follow robots.txt: OpenAI says rules may not apply to ChatGPT-User^{[1]Source 1: Overview of OpenAI crawlersOverview of OpenAI crawlersOpenAIaccessed: 26 September 2026}, and Perplexity says Perplexity-User generally ignores them^{[3]Source 3: Perplexity crawlersPerplexity crawlersPerplexityaccessed: 26 September 2026}.

See which bots your site allows with the AI bot access test.

3. Check your firewall

Even if robots.txt allows them, a firewall or bot protection can block unfamiliar requests. OpenAI and Perplexity publish their bots’ IP ranges as JSON files^{[1]Source 1: Overview of OpenAI crawlersOverview of OpenAI crawlersOpenAIaccessed: 26 September 2026[3]Source 3: Perplexity crawlersPerplexity crawlersPerplexityaccessed: 26 September 2026}; an allowlist can be built from them. Check your server logs for 403 or 429 responses to these bots.

4. Do not tie content to JavaScript

According to Vercel’s 2024 measurement, major AI crawlers do not run JavaScript^{[7]Source 7: The rise of the AI crawler (Dec 2024)The rise of the AI crawler (Dec 2024)Vercelaccessed: 26 September 2026}. Google also notes that not all bots can run JavaScript and recommends server-side or pre-rendered HTML^{[8]Source 8: Understand JavaScript SEO basicsUnderstand JavaScript SEO basicsGoogle Search Centralaccessed: 26 September 2026}. The check is simple: turn off JavaScript in your browser and open the page; if the main text, prices and contact details are missing, bots cannot see them either.

5. llms.txt: add it, but set expectations right

llms.txt is a proposed file at the site root that gives AI tools a summary of the site and its key pages in Markdown^{[10]Source 10: The /llms.txt fileThe /llms.txt filellmstxt.orgaccessed: 26 September 2026}. Google says you do not need llms.txt or special markup to appear in its AI features^{[9]Source 9: Optimizing your website for generative AI features on Google SearchOptimizing your website for generative AI features on Google SearchGoogle Search Centralaccessed: 26 September 2026}. It costs little, so you may add it; but it is not a ranking lever. Use the llms.txt generator to prepare one.

Checklist

• Search and answer bots are allowed in robots.txt.
• A deliberate decision has been made about training bots.
• Private paths are repeated in bot-specific groups.
• The firewall does not block these bots; IP lists are configured.
• The main text is readable with JavaScript off.
• Pages carry no noindex or unnecessary nosnippet.
• llms.txt (optional) follows the format.