Search for "AI crawler list" and you get dozens of pages that list 14, 44 or 67 bots. They mostly agree on the user agents. They disagree on almost everything that matters when you write a firewall rule: which bots publish IP addresses, which ones respect robots.txt, and which "bots" are only robots.txt tokens with no crawler behind them.
This list is built the other way round. We started from each operator's own documentation and IP files, fetched them, and wrote down only what they say. Every row links to its source. Everything was checked 12 Nov 2026.
It's a reference: who the bots are, what they're for, which robots.txt token controls them and whether the operator publishes their addresses. How to check a request against those addresses is a separate topic, covered step by step in how to verify AI crawler IP addresses.
How to read this list
AI bots fall into three groups, and you will usually want to treat them differently:
- Training crawlers collect pages that may be used to train models (GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot). Blocking them doesn't remove you from any AI search product.
- AI search crawlers build an index that AI answers cite (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot). Blocking them usually removes you from those answers.
- User-triggered fetchers load a page because a person asked an assistant to (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User). Several operators say robots.txt may not apply to them.
Then there are tokens: names you can write in robots.txt that don't crawl anything themselves. Google-Extended and Applebot-Extended are the two that matter.
For each bot, the useful question is not only "what's the user agent?" but "can I prove a request really came from it?" A user agent is a header the client picks. An IP address inside a range the operator publishes is much harder to fake. That's why every table below has a "Published IPs" column, the one most lists leave out.
The list
OpenAI
Source: OpenAI crawlers and user agents (checked 12 Nov 2026).
| Bot | Purpose | robots.txt | Published IPs |
|---|---|---|---|
| GPTBot | Crawls content that may be used to train OpenAI's foundation models | Respected; Disallow opts you out of training | gptbot.json, 18 IPv4 prefixes |
| OAI-SearchBot | Surfaces sites in ChatGPT search | Respected; opting out removes you from search answers | searchbot.json, 39 IPv4 prefixes |
| ChatGPT-User | User-initiated actions in ChatGPT and Custom GPTs | OpenAI: "robots.txt rules may not apply" | chatgpt-user.json, 237 IPv4 prefixes |
| OAI-AdsBot | Checks pages submitted as ads on ChatGPT; not used for training | Only visits pages submitted as ads | adsbot.json, 2 IPv4 prefixes |
GPTBot's current user agent ends in compatible; GPTBot/1.4; +https://openai.com/gptbot. OpenAI says the version can change, so match the token GPTBot, never the whole string. OpenAI also notes that search changes can take about 24 hours after a robots.txt edit. Details and verification code are in OpenAI crawler IP ranges explained.
Anthropic
Source: Anthropic's crawler help article (checked 12 Nov 2026).
| Bot | Purpose | robots.txt | Published IPs |
|---|---|---|---|
| ClaudeBot | Collects content that could contribute to model training | Respected, plus Crawl-delay | bots.json, 14 IPv4 prefixes, shared by all three bots |
| Claude-SearchBot | Improves search result quality | Respected | same file |
| Claude-User | Fetches pages when a Claude user asks | Has its own token; Anthropic notes disallowing it may reduce your visibility in user-directed search | same file |
Because the three bots share one file, the IP tells you "this is Anthropic", not which of the three it is. Use the user agent for that, after the IP has confirmed the operator. Anthropic also warns that blocking its IPs "may not work correctly or persistently guarantee an opt-out", because a blocked bot can't read your robots.txt. More in ClaudeBot and PerplexityBot IP addresses.
Perplexity
Source: Perplexity crawlers (checked 12 Nov 2026).
| Bot | Purpose | robots.txt | Published IPs |
|---|---|---|---|
| PerplexityBot | Indexes sites for Perplexity search; "not used to crawl content for AI foundation models" | Respected | perplexitybot.json, 13 IPv4 addresses |
| Perplexity-User | Fetches a page because a user asked | "generally ignores robots.txt rules" | perplexity-user.json, 10 IPv4 addresses |
Source: Google's common crawlers (checked 12 Nov 2026).
| Name | What it is | robots.txt | Published IPs |
|---|---|---|---|
| Googlebot | Google Search crawler | Respected | common-crawlers.json |
| Google-Extended | A token, not a crawler. Controls use of your content for Gemini training and grounding | User-agent: Google-Extended | None: it "doesn't have a separate HTTP request user agent string" |
| GoogleOther | Generic crawler used by product teams, for example for research | Respected | common-crawlers.json |
| Google-Agent and other user-triggered agents | Agents hosted on Google infrastructure acting for a user | See Google's fetcher docs | user-triggered-agents.json |
Google says Google-Extended "is not used as a ranking signal in Google Search". You can't enforce Google-Extended at the network level: the requests come from ordinary Googlebot addresses, so an IP block would also remove you from Search. To verify Googlebot itself, see Verify Googlebot and Bingbot.
Apple
Source: About Applebot (checked 12 Nov 2026).
| Name | What it is | robots.txt | Published IPs |
|---|---|---|---|
| Applebot | Powers Spotlight, Siri and Safari search; data may also help train Apple foundation models | Respected | applebot.json, 24 IPv4 prefixes; reverse DNS in *.applebot.apple.com |
| Applebot-Extended | A token. Opts you out of training Apple's foundation models. Apple: "Applebot-Extended does not crawl webpages" | User-agent: Applebot-Extended | None |
Amazon
Source: About Amazonbot (page last updated 26 Oct 2026, checked 12 Nov 2026).
| Bot | Purpose | robots.txt | Published IPs |
|---|---|---|---|
| Amazonbot | Improves Amazon products; "may be used to train Amazon AI models" | Respected (no Crawl-delay) | IP list page, 1.292 single addresses |
| Amzn-SearchBot | Search experiences such as Alexa; not used for generative AI training | Respected | IP list page, 816 addresses |
| Amzn-User | Live fetches for user requests | "may not follow all robots.txt directives" | IP list page, 1.023 addresses |
Amazon also honours a noarchive robots meta tag as "no use for model training". Several popular lists say Amazon publishes no ranges, or give a reverse DNS suffix to check. Neither matches Amazon's current page, which publishes the three lists above and says nothing about reverse DNS. We cover this in Amazonbot, Applebot, Bytespider and Meta crawlers.
Meta
Source: Meta web crawlers (checked 12 Nov 2026).
| Bot | Purpose | robots.txt | Published IPs |
|---|---|---|---|
| Meta-ExternalAgent | Training foundation AI models, or indexing content directly | Block it in robots.txt | None on the page |
| Meta-WebIndexer | Improves Meta AI search results | Block it in robots.txt | None on the page |
| Meta-ExternalFetcher | Fetches individual links at a user's request | May bypass robots.txt | None on the page |
| Meta-ExternalAds | Advertising and business products | Block it in robots.txt | None on the page |
| FacebookExternalHit | Link previews for shared content | May bypass robots.txt for security checks | None on the page |
Meta's page says allow-listing by IP address is "more secure" than by user agent, but it doesn't publish the addresses. Meta says its crawlers may cache robots.txt for up to 24 hours.
Common Crawl, Mistral, DuckDuckGo, Kimi, ByteDance
| Bot | Purpose | robots.txt | Published IPs | Source |
|---|---|---|---|---|
| CCBot | Builds Common Crawl's open repository of web crawl data, which many AI developers train on | Respected | ccbot.json, 4 IPv4 + 1 IPv6 prefix; IPv4 reverse DNS under crawl.commoncrawl.org | CCBot |
| MistralAI-Index | Index for Mistral search; not used for training | Respected | mistralai-index-ips.json, 2 addresses | Mistral robots |
| MistralAI-User | Fetches pages for user questions in Vibe | Governs which sites user requests can reach | mistralai-user-ips.json, 4 addresses | same |
| MistralAI-Training | Builds training datasets | Can be disallowed | None published | same |
| DuckAssistBot | Real-time pages for DuckDuckGo AI answers; "not used in any way to train AI models" | Respected (about three days to apply) | duckassistbot.json, 486 addresses, same as DuckDuckBot's list | DuckAssistBot |
| KimiBot | Content that may train Kimi's foundation models | Respected, Crawl-delay supported | kimibot.json, 4 addresses | Kimi crawlers |
| Kimi-SearchBot / Kimi-User | Search index / user-initiated retrieval | Kimi-User: rules "may not directly apply" | kimi-searchbot.json, kimi-user.json, 4 addresses each | same |
| Bytespider | Toutiao Search crawler (ByteDance) | Toutiao offers a crawl-rate setting | Ten /24 subnets listed on the page; hostnames *.bytedance.com | About Bytespider |
Two notes on that table:
- Bytespider is often described as ByteDance's AI training crawler. ByteDance's own page describes it as the Toutiao Search crawler and says nothing about AI. We describe it the way the operator does.
- Mistral publishes IPs for search and user fetches, not for training. You can disallow MistralAI-Training in robots.txt but you can't verify it by IP.
Who publishes what: the summary
| Can you verify it by IP? | Bots |
|---|---|
| Yes, machine-readable file | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot / Claude-SearchBot / Claude-User (shared), PerplexityBot, Perplexity-User, Googlebot and Google's agents, Applebot, CCBot, MistralAI-Index, MistralAI-User, DuckAssistBot, KimiBot, Kimi-SearchBot, Kimi-User |
| Yes, but the list is in an HTML page | Amazonbot, Amzn-SearchBot, Amzn-User (JSON inside the page), Bytespider (ten /24s in the text) |
| By reverse DNS | Applebot (applebot.apple.com), CCBot IPv4 (crawl.commoncrawl.org), Googlebot (googlebot.com, google.com) |
| No | Meta-ExternalAgent and the other Meta crawlers, MistralAI-Training |
| Nothing to verify: a token | Google-Extended, Applebot-Extended |
A robots.txt that matches common intent
Most sites want this: no training, yes to AI search, user fetches allowed. Here is that intent written out with the tokens above:
# Training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: KimiBot
Disallow: /
# AI search and user fetches: yes
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /Grouping several User-agent lines over one rule set is valid under the Robots Exclusion Protocol (RFC 9309). Some notes before you copy it:
- Amazonbot is a judgment call. Amazon says Amazonbot "may be used to train Amazon AI models" and also improves products such as Alexa answers. Amzn-SearchBot is the one that doesn't train.
- Applebot is not in the list on purpose. Disallowing Applebot removes you from Spotlight, Siri and Safari suggestions. Applebot-Extended is the training opt-out.
- Googlebot is not in the list either. Google-Extended is the training control; blocking Googlebot removes you from Search.
- Robots.txt is a request. It works for operators that honour it, which includes every training crawler above. It does nothing against scrapers that borrow a famous name.
From list to enforcement
Robots.txt states your intent. To enforce it, or to safely allow a bot that a firewall would otherwise challenge, you need to check each request's IP against the files in the "Published IPs" column. The method (downloading the files, matching in code, handling stale or failed downloads, combining the IP with the user agent) is explained once, in how to verify AI crawler IP addresses. For the operators that verify by reverse DNS instead, see verifying Googlebot and Bingbot.
Ready-made configurations for each stack:
- Nginx, generated
geoblocks - Cloudflare WAF, with lists synced through the API
- Next.js and Vercel, in
proxy.ts - Apache and HAProxy
- AWS WAF, IP sets synced from the files
If you'd rather not maintain the downloads, the IPAPI.GR lookup API returns risk.is_ai and risk.is_crawler for any IP. IPAPI.GR's risk flags tell you whether an address belongs to a known AI crawler, crawler or scanner, so you don't have to maintain the lists yourself. You can also check a single address from your logs with the free bot verification tool, or browse every covered bot on the bot signals page, with a page for each one, such as GPTBot or Amazonbot.
FAQ
Why do AI crawler lists disagree so much?
Partly because new bots appear every few months. Mostly because many lists copy each other instead of reading the operators' pages. Bytespider is a good example: several lists say ByteDance publishes no IP ranges, but Toutiao's own Bytespider page lists ten /24 subnets.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is the training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot won't appear in search answers, though they can still appear as navigational links.
Which AI bots ignore robots.txt?
None of the training crawlers above say they ignore it. The user-triggered fetchers are the exception: OpenAI says robots.txt "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores" it, Amazon says Amzn-User "may not follow all robots.txt directives", and Meta says Meta-ExternalFetcher may bypass it. If you need to control those, the IP is the handle.
Is Google-Extended a user agent I can block in my firewall?
No. It's only a robots.txt token. Google's crawling for Gemini uses the normal Google crawlers, so there's no separate traffic to block.
How do I check that a request really is GPTBot or ClaudeBot?
Match its IP against the operator's published file for that bot, and treat a famous user agent from any other address as an impostor. How to verify AI crawler IP addresses has the method with Node.js and Python code.
How often does this list change?
The operator files change more often than the bot names. On 12 Nov 2026, Google's files had been regenerated the day before and ChatGPT-User's a week before, while Mistral's user list still carried a February 2025 timestamp. Don't copy addresses from any article, including this one. Download the files.