Most “AI crawler lists” you’ll find are copied from each other, and many are out of date. They still list Claude-Web as Anthropic’s crawler, invent a “Googlebot-Extended” bot, or tell you Google-Extended crawls your pages. It doesn’t. Google-Extended has no user agent of its own; it’s only a robots.txt token.
This list is different in two ways. Every user agent below is checked against the operator’s own documentation, as of 20 November 2026. And we measured how real sites treat these bots: we fetched robots.txt from 3,000 randomly chosen sites in the Online Web Directory registry and counted which AI crawlers they name and block. You’ll also get tested robots.txt, Nginx and Apache rules, plus a script that checks whether a “GPTBot” visitor really comes from OpenAI.
What is an AI crawler list? An AI crawler list is a reference of the bots that AI companies use to fetch web pages, with each bot’s robots.txt token, its user-agent string and its job: collecting training data, indexing pages for AI search, or fetching a page live because a user asked. You use it to write robots.txt rules, read server logs and set firewall policies.
AI crawlers at a glance
| Question | Short answer |
|---|---|
| How many AI crawlers matter? | About 20 documented user agents from 9 operators (OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Common Crawl, Mistral), plus DuckDuckGo and a long tail. |
| Main types | Training crawlers, AI search crawlers, and user-triggered fetchers or agents. Google-Extended and Applebot-Extended are control tokens, not crawlers. |
| Do they obey robots.txt? | Documented training and search crawlers say they do. User-triggered fetchers (ChatGPT-User, Perplexity-User, Google’s user-triggered fetchers, Meta-ExternalFetcher) say robots.txt may not apply. |
| Can robots.txt return a 403? | No. robots.txt is a request, not a lock. A 403 comes from your server, CDN or firewall. |
| How often do sites block them? | In our 20 Nov 2026 sample, 16.9% of robots.txt files fully block GPTBot but only 2.8% fully block OAI-SearchBot. |
| How do I verify a bot? | Match the request IP against the operator’s published IP list (for example https://openai.com/gptbot.json). User agents are trivial to fake. |
The AI crawler list (verified November 2026)
The table below lists each robots.txt token, who runs it, what it does according to the operator, and whether the operator says it follows robots.txt. “Type” uses three buckets: training (collects content that may train models), search (indexes pages so an AI product can cite them later) and user (fetches a page in real time because a person asked). We left out bots whose operator publishes no documentation we could verify.
| robots.txt token | Operator | Type | What it does | Follows robots.txt? |
|---|---|---|---|---|
GPTBot |
OpenAI | Training | Crawls content that may be used to train OpenAI’s foundation models. | Yes |
OAI-SearchBot |
OpenAI | Search | Indexes pages so they can appear in ChatGPT search results. | Yes |
ChatGPT-User |
OpenAI | User | Visits a page when a ChatGPT or Custom GPT user asks a question. | “May not apply” |
OAI-AdsBot |
OpenAI | Ads check | Checks landing pages submitted as ChatGPT ads; not used for training. | Only visits submitted pages |
ClaudeBot |
Anthropic | Training | Collects web content that could contribute to training Claude models. | Yes, and honours Crawl-delay |
Claude-SearchBot |
Anthropic | Search | Indexes content to improve Claude’s search results. | Yes |
Claude-User |
Anthropic | User | Fetches pages when a Claude user asks a question. | Yes, per Anthropic |
Google-Extended |
Control token | Decides whether content Google crawls can train Gemini or ground Gemini answers. No separate crawler. | Token only | |
Google-CloudVertexBot |
Owner-requested | Crawls sites whose owners build Vertex AI Agents; no effect on Google Search. | Yes | |
Google-Agent |
User | Agents on Google infrastructure that browse and act on a user’s request. | User-triggered: generally no | |
PerplexityBot |
Perplexity | Search | Indexes pages for Perplexity search results; not used for training. | Yes |
Perplexity-User |
Perplexity | User | Fetches a page during a user’s Perplexity session. | “Generally ignores” |
Applebot-Extended |
Apple | Control token | Decides whether pages Applebot crawled can train Apple’s foundation models. Doesn’t crawl. | Token only |
meta-externalagent |
Meta | Training | Crawls for uses such as training AI models or indexing content. | Yes |
meta-externalfetcher |
Meta | User | Fetches individual links at a user’s request, including agentic tasks. | “May bypass” |
Amazonbot |
Amazon | Training | Improves Amazon products and services; may be used to train Amazon AI models. | Yes |
Amzn-SearchBot |
Amazon | Search | Indexes pages for search experiences such as Alexa; not used for training. | Yes |
Amzn-User |
Amazon | User | Fetches live information to answer a user, for example an Alexa query. | “May not follow all” |
CCBot |
Common Crawl | Training data | Builds the open Common Crawl web archive, a common source of AI training data. | Yes |
MistralAI-Training |
Mistral AI | Training | Crawls content to build datasets for training Mistral models. | Yes |
MistralAI-Index |
Mistral AI | Search | Indexes content for Mistral search; not used for training. | Yes |
MistralAI-User |
Mistral AI | User | Visits a page when a user asks Mistral’s Vibe assistant a question. | Yes, for user requests |
DuckAssistBot |
DuckDuckGo | Search | Crawls pages in real time for DuckDuckGo’s AI-assisted answers. | Yes (takes effect after 72 hours) |
Sources for every row are linked at the end of this article. Quotes in the last column come straight from the operator’s documentation. Bytespider (ByteDance) is missing on purpose: we couldn’t find a ByteDance page that documents it, even though 18.5% of the robots.txt files we sampled name it. For the same reason we don’t describe the long tail of tokens such as cohere-ai, Diffbot or omgili; you’ll find how often sites name them in the data section.
Full user-agent strings you’ll see in logs
robots.txt matches on the short token, but your access logs show the full string. These are the examples the operators publish. Version numbers change, so match on the token, not the whole string.
# GPTBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
# OAI-SearchBot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
# ChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
# PerplexityBot
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
# Perplexity-User
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
# Amazonbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36
# CCBot
CCBot/2.0 (https://commoncrawl.org/faq/)
# meta-externalagent
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
# MistralAI-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
# DuckAssistBot
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
OpenAI also notes that GPTBot and OAI-SearchBot may add a robots.txt marker to the user agent when they fetch your robots.txt file, so you can tell those requests apart in logs that don’t record paths.
Three kinds of AI crawler, and why the difference matters
Blocking “AI bots” as one group is the most common mistake we see. The three types have different costs and benefits, and operators treat robots.txt differently for each.
Training crawlers
GPTBot, ClaudeBot, CCBot, meta-externalagent, Amazonbot and MistralAI-Training collect pages that may end up in model training data. Blocking them doesn’t remove you from AI search results run by the same company: OpenAI says each of its robots.txt settings is independent, so you can disallow GPTBot and still allow OAI-SearchBot. In our sample, training crawlers are the most blocked group. 17.3% of robots.txt files fully disallow CCBot, 16.9% GPTBot and 15.9% ClaudeBot.
AI search crawlers
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, MistralAI-Index and DuckAssistBot index pages so an assistant can find and cite them later. Blocking them can keep you out of those answers. Site owners seem to know this: only 2.8% of files fully block OAI-SearchBot and 5.8% block PerplexityBot, against 16.9% for GPTBot.
User-triggered fetchers and agents
ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User, MistralAI-User and Google-Agent act for one person at a time, like a browser would. Several operators say robots.txt may not apply because a human asked for the page. These are also the bots that shopping and booking agents use, so blocking them can turn away real customers. 5.9% of files in our sample fully block ChatGPT-User.
Control tokens that don’t crawl
Google-Extended and Applebot-Extended never visit your site. Google and Apple crawl with Googlebot and Applebot as usual, then check these tokens to decide whether the content can train their models. Disallowing Google-Extended doesn’t affect Google Search inclusion or ranking, according to Google. 14.1% of sampled files fully block Google-Extended and 13.5% Applebot-Extended. Google has also said it plans to support the shared AI-training signal (user-agent: __usage_ai-training) in early 2027; we cover it in our guide to the robots.txt AI training opt-out.
Legacy tokens that still fill robots.txt files
Anthropic’s current help page lists ClaudeBot, Claude-User and Claude-SearchBot only. Yet 15.6% of sampled files still name anthropic-ai and 13.8% name Claude-Web. Rules for those names are harmless, but if they’re your only Anthropic rules, they may not cover the bots Anthropic documents today. Check that ClaudeBot is named explicitly.
What AI crawler access looks like in practice
Operator documentation is the source of truth for user agents. OpenAI’s page, for example, lists each bot with an example user-agent string, notes on the robots.txt marker and a link to its published IP file.

What a site actually does with these bots is a different matter. We requested six home pages from our server with a Chrome user agent, OpenAI’s GPTBot user agent and a ClaudeBot user agent. Our server sits on a datacenter IP, which many bot defences treat with suspicion, so results from a home connection can differ.

Three patterns show up. Reddit served Chrome but returned 403 to GPTBot and 429 to ClaudeBot, which is user-agent filtering at the server or CDN. The New York Times returned 403 to all three, which suggests the block was based on our IP or the request’s fingerprint, not the user agent. And bbc.com returned 200 to GPTBot even though its robots.txt disallows GPTBot, ClaudeBot, CCBot, PerplexityBot and at least five other AI tokens. That’s the clearest proof that robots.txt asks and doesn’t enforce. Amazon answered all three with a 202 and an empty body, an interstitial challenge rather than a page.
The Online Web Directory record for each site keeps these two layers apart: whether agents can reach the site, and what its robots.txt says.

How to check which AI crawlers can reach your site
1. Read your robots.txt groups for AI tokens
Start with what you’re asking crawlers to do. This command prints every group that names a common AI token, with the line after it:
curl -s https://www.bbc.com/robots.txt | grep -i -A1 -E '^user-agent: *(gptbot|oai-searchbot|chatgpt-user|claudebot|google-extended|perplexitybot|ccbot|bytespider|applebot-extended)'
Real output for bbc.com on 20 November 2026 at 08:09 AEDT (trimmed to four of the nine groups):
User-agent: CCBot
Disallow: /
--
User-agent: ClaudeBot
Disallow: /
--
User-agent: GPTBot
Disallow: /
--
User-agent: OAI-SearchBot
Disallow: /
Remember that a crawler obeys only the most specific group that matches its token. If you name GPTBot in its own group, it ignores your User-agent: * rules entirely, so repeat any shared rules inside the GPTBot group.
2. Test how your server responds to AI user agents
robots.txt can say “no” while your server still says “yes”, or the reverse. Request the same page with different user agents and compare status codes:
GPT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'
CHROME='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36'
for ua in "$CHROME" "$GPT"; do
curl -s -o /dev/null -A "$ua" -w "%{http_code} %{size_download}B\n" https://www.example-shop.com/
done
Replace the URL with your own. A 403 or 429 for GPTBot alone means user-agent filtering. A 403 for both usually means IP reputation or a bot challenge. Run it from a home connection as well as a server: real crawlers come from datacenter IPs, so the server result is closer to what they see.
3. Find AI crawlers in your access logs
Count hits by bot, then check which status codes each bot received. We ran these against the access log of a test Nginx server on 20 November 2026:
$ grep -o -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|meta-external[a-z]+' access.log | sort | uniq -c | sort -rn
2 OAI-SearchBot
2 GPTBot
$ awk -F'"' '/GPTBot/ {split($3,a," "); print a[1]}' access.log | sort | uniq -c
1 200
1 403
The second command works on the standard “combined” log format, where the status code follows the quoted request line. A GPTBot 200 on /robots.txt plus a 403 on everything else is exactly what you want when you block training crawlers at the server.
4. Verify that a “GPTBot” visitor is really OpenAI
Anyone can send a GPTBot user agent. OpenAI, Perplexity, DuckDuckGo, Mistral and Google publish IP lists, so check the source IP. This script tests an IP against OpenAI’s three files:
# verify_ip.py - usage: python3 verify_ip.py 132.196.86.1
import sys, json, ipaddress, urllib.request
ip = ipaddress.ip_address(sys.argv[1])
for name in ["gptbot", "searchbot", "chatgpt-user"]:
data = json.load(urllib.request.urlopen(f"https://openai.com/{name}.json", timeout=20))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
hit = [str(n) for n in nets if ip.version == n.version and ip in n]
print(f"{name:13} {len(nets):3} ranges {'MATCH ' + hit[0] if hit else 'no match'}")
Output on 20 November 2026 at 08:12 AEDT:
$ python3 verify_ip.py 132.196.86.1
gptbot 18 ranges MATCH 132.196.86.0/24
searchbot 39 ranges no match
chatgpt-user 237 ranges no match
$ python3 verify_ip.py 1.1.1.1
gptbot 18 ranges no match
searchbot 39 ranges no match
chatgpt-user 237 ranges no match
At the time of our test, OpenAI’s gptbot.json held 18 IPv4 ranges with a creation time of 22 September 2026. Fetch the files fresh rather than hard-coding them.
5. Look up your site in the registry
Online Web Directory checks how sites respond to plain HTTP clients and what their robots.txt says. Search for your domain, for example reddit.com or nytimes.com, or run the free readiness check. The sites that block AI agents list shows which domains turn plain clients away.

How to control AI crawlers: six methods
Pick based on what you want. Most sites want to keep AI search traffic, decide case by case on training, and leave user-triggered agents alone.
Method 1: Block training crawlers but keep AI search (robots.txt)
This is the most common policy among the sites we sampled. Add this to /robots.txt at the root of every host you serve:
# Block AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /
# Allow AI search crawlers (so you can be cited in AI answers)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
Allow: /
# Everyone else
User-agent: *
Disallow: /wp-admin/
We tested this file with Protego, an RFC 9309 parser. Results for the path /blog/:
GPTBot /blog/ blocked
ClaudeBot /blog/ blocked
CCBot /blog/ blocked
Google-Extended /blog/ blocked
Applebot-Extended /blog/ blocked
OAI-SearchBot /blog/ allowed
Claude-SearchBot /blog/ allowed
PerplexityBot /blog/ allowed
Googlebot /blog/ allowed
Bingbot /blog/ allowed
Stacking several User-agent lines over one set of rules is valid under RFC 9309. Note that the AI search group doesn’t inherit the /wp-admin/ rule from the * group; add it there too if you need it.
Method 2: Block every documented AI crawler (robots.txt)
If you don’t want any AI use, add the search and user tokens to the disallow group: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, meta-externalfetcher, Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-User and DuckAssistBot. Expect to disappear from AI answers. And remember that several user-triggered fetchers say robots.txt may not apply to them, so this is a statement of preference, not a wall.
Method 3: Enforce the block in Nginx
To actually refuse requests, filter at the server. This config returns 403 to training crawlers on every path except robots.txt, so they can still read your rules. The map block goes in the http context:
# Block AI training crawlers by user agent, but still let them read robots.txt
map $http_user_agent $block_ai_crawler {
default 0;
"~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent|Amazonbot|MistralAI-Training)" 1;
}
server {
listen 80;
root /var/www/html;
location = /robots.txt { }
location / {
if ($block_ai_crawler) { return 403; }
}
}
We tested it with Nginx 1.26.3 on 20 November 2026: nginx -t passed, GPTBot got 403 on / and 200 on /robots.txt, while Chrome and OAI-SearchBot got 200 on both. A single return inside if is one of the uses Nginx documents as safe.
Method 4: Enforce the block in Apache (.htaccess)
The same rule for Apache with mod_rewrite:
# Block AI training crawlers by user agent, but still let them read robots.txt
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent|Amazonbot|MistralAI-Training) [NC]
RewriteRule ^ - [F,L]
</IfModule>
Tested on Apache 2.4.71 with AllowOverride All: GPTBot received 403 on the home page and 200 on robots.txt; Chrome and OAI-SearchBot received 200. The [F] flag sends 403 Forbidden.
Method 5: Use your CDN’s AI crawler policies
If you’re on Cloudflare, you don’t need to maintain a list. Under Security, Settings, filter by “Bot traffic” and open AI Crawler and Agent access policies. Cloudflare lets you set separate actions for Search, Agent and Training traffic. Its docs say new domains default to “Allow (do not block)” for all three, and that “Disallow” for Training only edits robots.txt unless Bot Preference Sync is on. Cloudflare’s AI Crawl Control dashboard also shows which crawlers visit and whether they follow your robots.txt. Of the 62,088 sites in our registry, 6,966 (11.2%) showed a Cloudflare challenge to our checker.
Method 6: Verify, then allow
User-agent rules block honest bots and miss dishonest ones. If you block at the edge, pair the rule with an IP check (see step 4 above) and allow verified search and user agents through. Anthropic warns that blocking its IP addresses may not work reliably as an opt-out, because it stops ClaudeBot from reading your robots.txt. Use robots.txt for the policy and IP checks to catch impostors.
What 2,124 robots.txt files say about AI crawlers
On 20 November 2026 between 08:01 and 08:03 AEDT, we picked 3,000 hosts at random from the 62,088 sites in the Online Web Directory open dataset and fetched /robots.txt from our server. 2,124 returned a parseable file. A token counts as “named” if it has its own User-agent line, and as “fully blocked” if that group has Disallow: / without Allow: /.
| robots.txt token | Named | Fully blocked |
|---|---|---|
GPTBot |
23.6% | 16.9% |
ClaudeBot |
22.0% | 15.9% |
CCBot |
20.6% | 17.3% |
Google-Extended |
19.4% | 14.1% |
Bytespider |
18.5% | 16.6% |
Amazonbot |
17.4% | 14.3% |
meta-externalagent |
16.7% | 14.0% |
Applebot-Extended |
16.6% | 13.5% |
anthropic-ai (legacy) |
15.6% | 13.3% |
Claude-Web (legacy) |
13.8% | 12.2% |
PerplexityBot |
12.4% | 5.8% |
ChatGPT-User |
12.2% | 5.9% |
OAI-SearchBot |
9.3% | 2.8% |
Claude-User |
6.5% | 3.2% |
Perplexity-User |
6.4% | 3.2% |
Claude-SearchBot |
6.2% | 2.8% |
DuckAssistBot |
4.2% | 2.7% |
MistralAI-User |
3.7% | 2.6% |
Four findings stand out:
- Most sites don’t mention AI crawlers at all. 629 of 2,124 files (29.6%) name at least one AI-specific token. Those that do tend to name many: the median is 12 tokens, and 357 files name 10 or more, a sign of copy-pasted blocklists.
- Training gets blocked, search doesn’t. GPTBot is fully blocked six times as often as OAI-SearchBot (16.9% against 2.8%).
- robots.txt and real access disagree. Of the 358 sites that fully block GPTBot in robots.txt, 284 (79%) were “open” in OWD’s own access check, meaning nothing at the server stopped a plain client.
- Blanket bans are rare. Only 13 files (0.6%) disallow everything for all user agents.
The wider registry tells the same story from the other side. These are the access results for all 62,088 sites, last checked between 11 October and 19 November 2026:
| What a plain HTTP client met | Sites | Share |
|---|---|---|
| Open | 50,844 | 81.9% |
| Bot challenge | 7,821 | 12.6% |
| Blocked | 3,060 | 4.9% |
| Rate limited | 199 | 0.3% |
| Login required | 150 | 0.2% |
| Paywalled | 14 | 0.0% |
Download the full dataset (CC BY 4.0) from onlinewebdirectory.net/exports/sites.csv, or see the AI agent access report.
Related errors and responses
- 403 Forbidden. Your server, CDN or firewall refused the request. It’s never caused by robots.txt itself, and it isn’t permanent: change the rule and the next request can succeed.
- 429 Too Many Requests. Rate limiting. Reddit returned this to our ClaudeBot request. Anthropic supports
Crawl-delay, a gentler alternative for ClaudeBot. - 202 with an empty body, or a 503 challenge page. Usually a JavaScript or CAPTCHA challenge. Crawlers can’t solve these, so to them the page is blocked even though no 403 was sent.
- robots.txt returns 4xx. Under RFC 9309, a 400-499 response means the file is “unavailable” and crawlers may access everything. In our sample, 314 of 3,000 sites (10.5%) answered our robots.txt request with 403, which a compliant crawler would read as no rules at all.
- robots.txt returns 5xx or times out. RFC 9309 says crawlers must then assume complete disallow, so an outage on robots.txt can quietly block every crawler, including Googlebot.
- robots.txt larger than 500 KiB. RFC 9309 only requires crawlers to parse the first 500 KiB, so rules past that point may be ignored.
If you publish an llms.txt file for AI agents, make sure the bots you allow can actually fetch it. Our llms.txt guide shows how to test that.
Summary
An accurate AI crawler list starts with the operators’ own documentation, not other lists. Today that means about 20 user agents from 9 companies, in three types: training crawlers, AI search crawlers and user-triggered agents, plus two control tokens (Google-Extended and Applebot-Extended) that never crawl. Most sites that act on this block training and keep search, and robots.txt is only a request: if you need a real block, enforce it at the server or CDN and verify IPs.
AI agents already look sites up in the Online Web Directory before they visit. If your site’s record is wrong, or you want agents to know your API, MCP server or AI permissions, claim your site for $25. You can run the free readiness check first.
Frequently asked questions
What is the user agent for ChatGPT?
ChatGPT uses three. GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a user asks. Their robots.txt tokens are GPTBot, OAI-SearchBot and ChatGPT-User.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI controls search through OAI-SearchBot. You can disallow GPTBot to opt out of training and still allow OAI-SearchBot so your pages can appear and be cited in ChatGPT search.
Is Google-Extended a crawler?
No. Google says Google-Extended has no separate user agent. Googlebot does the crawling, and the Google-Extended token only controls whether that content can be used for Gemini training and grounding. It doesn’t affect Google Search ranking.
Do AI crawlers follow robots.txt?
The documented training and search crawlers in this list say they do. User-triggered fetchers such as ChatGPT-User, Perplexity-User and Meta-ExternalFetcher say robots.txt may not apply, because a person requested the page. Undocumented scrapers may ignore it entirely.
How do I know if a bot is really GPTBot or ClaudeBot?
Check the source IP. OpenAI publishes ranges in gptbot.json, searchbot.json and chatgpt-user.json; Perplexity, DuckDuckGo, Mistral and Google publish their own. A matching user agent from an unlisted IP is an impostor.
Sources
- OpenAI, Overview of OpenAI crawlers (accessed 20 Nov 2026)
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google, Google’s common crawlers and user-triggered fetchers
- Perplexity, Perplexity crawlers
- Apple, About Applebot
- Meta, Meta web crawlers
- Amazon, Amazonbot and Amazon crawlers
- Common Crawl, CCBot
- Mistral AI, Mistral AI robots
- DuckDuckGo, About DuckAssistBot
- IETF, RFC 9309: Robots Exclusion Protocol
- Cloudflare, AI Crawler and Agent access policies and AI Crawl Control
- Online Web Directory, open site dataset (62,088 sites, exported 20 Nov 2026) and our robots.txt sample of 3,000 hosts

