Engines · Meta AI
Visibility in Meta AI: Meta’s bots and robots.txt decisions
Meta runs several bots that visit the web, and they do different jobs: one builds previews of shared links, one improves Meta AI’s search results, one collects content for model training. This page explains which does what, from Meta’s own documentation.
The bot related to Meta AI’s search results is Meta-WebIndexer, and Meta recommends allowing it. Meta-ExternalAgent crawls for foundation model training and product improvement. Both respect robots.txt. Meta-ExternalFetcher fetches links at a user’s request and may bypass robots.txt. You can close training and stay visible in search.
Which Meta bots visit your site?
| Bot | Purpose | robots.txt |
|---|---|---|
| Meta-WebIndexer | Navigates the web to improve Meta AI search result qualitySource 1: Meta Web Crawlers | Respected; Meta recommends allowing itSource 1: Meta Web Crawlers. |
| Meta-ExternalAgent | Crawls for uses such as training foundation AI models or improving productsSource 1: Meta Web Crawlers | RespectedSource 1: Meta Web Crawlers. |
| Meta-ExternalFetcher | Fetches individual links at a user’s request; also supports improving agentic capabilitiesSource 1: Meta Web Crawlers | May bypass itSource 1: Meta Web Crawlers. |
| FacebookExternalHit | Crawls the content of links shared in Meta’s appsSource 1: Meta Web Crawlers | May bypass it for security or integrity checksSource 1: Meta Web Crawlers. |
| Meta-ExternalAds | Crawls to improve advertising and other business productsSource 1: Meta Web Crawlers | RespectedSource 1: Meta Web Crawlers. |
To close training and stay visible in Meta AI’s search results, block Meta-ExternalAgent and allow Meta-WebIndexer. Our readiness test shows whether your site is open to these two bots.
How do we measure Meta AI?
- The question set is asked the same way every month, each question twice.
- We record whether the brand is mentioned, which sources are shown and whether the information is correct, with the date.
- Meta’s documentation does not detail source selection, so we see which sites are cited by measuring.
Frequently asked questions
01If I block Meta-ExternalAgent, do I disappear from Meta AI?
Meta-ExternalAgent is for training and product improvement; the bot related to Meta AI’s search results is Meta-WebIndexerSource 1: Meta Web Crawlers. You can block one and allow the other.
02Will previews of shared links break?
Previews are built by FacebookExternalHitSource 1: Meta Web Crawlers. If your rules for AI bots do not cover that bot, previews are not affected.
Sources
- [1]Meta Web CrawlersMeta for Developers · accessed: 26 September 2026
How to cite this page
Kaynaq. (26 September 2026). Visibility in Meta AI: Meta’s bots and robots.txt decisions. https://kaynaq.pages.dev/en/ai-engines/meta-ai
Updates and corrections
No corrections on this page yet. If you spot an error, write to us; we publish every fix here with its date.