Which AI crawlers should you allow?
Allow the crawlers that fetch pages for AI answers: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, and Googlebot and Bingbot for Google's and Microsoft's AI. Blocking training crawlers such as GPTBot and ClaudeBot keeps your pages out of training, not out of those answers.
Checked against each vendor's own docs on October 8, 2026.
The main AI crawlers, by the job each does.
AI companies split their crawling by job: one crawler builds a search index, another fetches a page when a user asks about it, a third collects training data. Each answers to its own name in robots.txt, so you can let one in and keep another out.[1, 2, 3] A linked name opens that crawler's page, with its user agent string and IP ranges.
| Crawler | What it does | If you block it | Cited sites blocking it |
|---|---|---|---|
| Search: decide whether AI answers can cite you | |||
| OAI-SearchBotOpenAI | What it doesFinds pages for ChatGPT search. | If you block itYour pages stop appearing in ChatGPT's search answers; they can still show as plain links.[1] | Sites blocking0.9%13 of 1,529 |
| Claude-SearchBotAnthropic | What it doesIndexes pages to improve Claude's search results. | If you block itClaude stops indexing your pages for search.[2] | Sites blocking0.8%12 of 1,529 |
| PerplexityBotPerplexity | What it doesFinds and links pages in Perplexity's search results. It doesn't collect training data. | If you block itPerplexity asks you to allow it if you want your site in its results.[3] | Sites blocking1.2%19 of 1,529 |
| GooglebotGoogle | What it doesCrawls for Google Search, AI Overviews and AI Mode included. | If you block itGoogle can't crawl your pages, and its AI features show only pages that can appear in Search with a snippet. To limit only what they quote, use nosnippet or max-snippet.[4, 5] | Sites blocking0.1%1 of 1,529 |
| BingbotMicrosoft | What it doesCrawls for Bing and Copilot, which share one index. | If you block itBing can't crawl your pages for search or Copilot, and lists blocking it as a practice to avoid. The noarchive tag keeps pages out of Copilot's answers instead.[6] | Sites blocking0.1%1 of 1,529 |
| ApplebotApple | What it doesCrawls for search in Siri and Spotlight. Apple may also train its models on what it fetches. | If you block itYour content can't appear in Apple's search results.[7] | Sites blocking0.7%10 of 1,529 |
| Meta-WebIndexerMeta | What it doesIndexes pages to improve Meta AI's search results. | If you block itMeta asks you to allow it if you want Meta AI to cite and link to your content.[8] | Sites blockingNot measured |
| Amzn-SearchBotAmazon | What it doesCrawls so your content can appear in search experiences such as Alexa. It doesn't collect training data. | If you block itYour content can't appear in those search experiences.[9] | Sites blockingNot measured |
| DuckAssistBotDuckDuckGo | What it doesFetches pages for DuckDuckGo's AI-assisted answers. It doesn't collect training data. | If you block itTakes effect after 72 hours and doesn't change your place in DuckDuckGo's results.[10] | Sites blockingNot measured |
| User fetches: read a page someone asked about | |||
| ChatGPT-UserOpenAI | What it doesVisits a page when a ChatGPT user's question calls for it. | If you block itOpenAI says robots.txt rules may not apply to it, since a person asked.[1] | Sites blocking1.3%20 of 1,529 |
| Claude-UserAnthropic | What it doesVisits a page when someone asks Claude a question. | If you block itClaude can't fetch your pages for people's questions. Anthropic says all its bots follow robots.txt.[2] | Sites blocking1.0%15 of 1,529 |
| Perplexity-UserPerplexity | What it doesVisits a page when a Perplexity user's question calls for it. | If you block itPerplexity says it generally ignores robots.txt, since a person asked.[3] | Sites blocking1.0%16 of 1,529 |
| Training: collect pages for future models | |||
| GPTBotOpenAI | What it doesCollects pages that may train OpenAI's models. | If you block itTells OpenAI not to train on your pages. ChatGPT search doesn't change.[1] | Sites blocking3.1%47 of 1,529 |
| ClaudeBotAnthropic | What it doesCollects pages that could train Anthropic's models. | If you block itYour future pages stay out of Anthropic's training data.[2] | Sites blocking2.6%40 of 1,529 |
| Google-ExtendedGoogle | What it doesA token, not a crawler: it covers Gemini's training, and grounding in the Gemini app and Vertex AI, on pages Googlebot fetched. | If you block itYour pages leave Gemini's training and the Gemini app's grounding. Google Search, AI Overviews included, doesn't change.[4] | Sites blocking2.0%31 of 1,529 |
| Applebot-ExtendedApple | What it doesA token, not a crawler: it covers training Apple's foundation models on pages Applebot fetched. | If you block itYour pages leave Apple's model training and stay in its search results.[7] | Sites blockingNot measured |
| CCBotCommon Crawl | What it doesBuilds Common Crawl's open archive of the web. OpenAI trained GPT-3 mostly on it. | If you block itCommon Crawl stops crawling your site.[11, 12] | Sites blocking4.2%64 of 1,529 |
| meta-externalagentMeta | What it doesCrawls for uses such as training Meta's foundation AI models. | If you block itMeta's crawler stops, after up to 24 hours on its cached copy of your robots.txt.[8] | Sites blockingNot measured |
| AmazonbotAmazon | What it doesCrawls pages that may train Amazon's AI models. | If you block itEach Amazon crawler follows its own rules, so Amzn-SearchBot, for search in Alexa, isn't affected.[9] | Sites blockingNot measured |
The sites AI cites keep search crawlers in.
We read the robots.txt of every site our recorded AI answers cited. 4.8% block at least one training crawler; 1.5% block OAI-SearchBot, Claude-SearchBot or PerplexityBot.
of cited sites block GPTBot, OpenAI's training crawler. 0.9% block OAI-SearchBot, which ChatGPT search reads through.
sites that block a training crawler still let OAI-SearchBot, Claude-SearchBot and PerplexityBot in.
site of 1,529 closes its homepage to Googlebot, and 1 to Bingbot.
Blue: crawlers that fetch for answers. Gray: training crawlers.
Five rules that don't fit the pattern.
Google-Extended reaches the Gemini app
Blocking it keeps your pages out of Gemini's training and out of the grounding the Gemini app uses for its answers. Google Search, AI Overviews and AI Mode included, follows Googlebot's rules alone.[4, 5]
User fetchers may skip robots.txt
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, since a person asked for the page. Anthropic says Claude-User follows robots.txt.[1, 2, 3] To stop a fetcher for good, block it at your firewall.
Applebot does search and training
Apple may train its models on what Applebot fetches for Siri and Spotlight. Disallow
Applebot-Extendedto opt out of training and stay in search; thenosnippettag keeps a page out of the world knowledge answers Apple's AI gives.[7]Bing has tags for Copilot
Bing and Copilot share one crawl and one index, so blocking Bingbot affects both. The
noarchivetag keeps your pages out of Copilot's answers while they stay in Bing's results, andnocachelimits Copilot to their URL, title and snippet.[6]One crawl can feed two jobs
If you allow both OAI-SearchBot and GPTBot, OpenAI may use one crawl for search and training. Disallowing GPTBot still keeps your pages out of training.[1]
You want AI answers to cite you
Allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, Meta-WebIndexer, Amzn-SearchBot and DuckAssistBot. With no robots.txt, or one that doesn't block them, they're allowed already.
You don't want your pages in training
Disallow GPTBot, ClaudeBot, CCBot, Applebot-Extended, meta-externalagent and Amazonbot. Add Google-Extended only if leaving the Gemini app's grounding is fine with you.
You want out of AI entirely
Keep Googlebot and Bingbot unless you can do without Google and Bing as well. Disallow the rest of the table, and block the user fetchers at your firewall, since robots.txt may not stop them.
You don't know what your site allows
The free AI crawler checker reads your robots.txt for 13 crawlers and fetches your homepage as 4 of them, so a firewall that turns them away shows up too.
Questions about AI crawlers.
Does blocking training crawlers take my site out of AI answers?
Not for GPTBot and ClaudeBot, which collect pages for model training. ChatGPT search reads pages through OAI-SearchBot and Claude's search through Claude-SearchBot, so those answers don't change. Google-Extended is the exception: blocking it also takes your pages out of the Gemini app's grounding. GPTBot, and what blocking it changes
Does blocking AI crawlers hurt my Google rankings?
Not unless you block Googlebot. Google says Google-Extended doesn't affect a site's inclusion in Google Search, and GPTBot, ClaudeBot and CCBot have nothing to do with Google.
Which crawlers does ChatGPT use to read my site?
Two. OAI-SearchBot builds the index ChatGPT search draws on, and ChatGPT-User visits a page when a user's question calls for it. GPTBot only collects training data.
Do AI crawlers follow robots.txt?
The search and training crawlers in this guide say they do. OpenAI and Perplexity say their user-requested fetchers may not, since a person asked; Anthropic says Claude-User does. Only a firewall rule stops a crawler that doesn't. Firewalls and AI crawlers
How many websites block AI crawlers?
Of the 1,529 sites in our study whose robots.txt we could read, or that have none, 3.1% block GPTBot, 2.6% ClaudeBot and 4.2% CCBot, against 0.9% for OAI-SearchBot and 0.1% for Googlebot. AI answers about AI visibility tools and web analytics cited these sites, so sites that block AI search crawlers may be rarer here, and other fields may differ.
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them the vendor's own documentation, read on October 8, 2026. The answers above repeat facts marked in the guide. Vendors change these pages, so check them before you act.
- 1OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- 2Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 3Perplexity: Perplexity Crawlersdocs.perplexity.ai/docs/resources/perplexity-crawlers
- 4Google: Google's common crawlersdevelopers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- 5Google: AI Features and Your Websitedevelopers.google.com/search/docs/appearance/ai-features
- 6Microsoft: Bing Webmaster Guidelinesbing.com/webmasters/help/webmaster-guidelines-30fba23a
- 7Apple: About Applebotsupport.apple.com/en-us/119829
- 8Meta: Meta web crawlersdevelopers.facebook.com/documentation/sharing/webmasters/web-crawlers
- 9Amazon: About Amazonbotdeveloper.amazon.com/amazonbot
- 10DuckDuckGo: Is DuckAssistBot related to DuckDuckGo?duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- 11Common Crawl: CCBotcommoncrawl.org/ccbot
- 12OpenAI: Language Models are Few-Shot Learners (Table 2.2)arxiv.org/abs/2005.14165
Keep reading
- The free AI crawler checkerTest 13 AI crawlers, from GPTBot to PerplexityBot, against your robots.txt, and fetch your homepage as 4 of them. Free, no sign-up.
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI visibility tools, comparedEvery tool we have checked, by starting price.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
- What is GEOGEO, generative engine optimization, in plain words: how AI answers pick their sources, what research shows helps, and how to measure your AI visibility.
- AEO vs GEO vs SEOAEO vs GEO vs SEO: where each name came from, what Google and Microsoft say, the work all three share, and the four things that change with AI answers.
- Check if ChatGPT mentions your brandCheck by hand in five steps, see why one chat can't measure it (answers change, memory, location, leading questions), and where ChatGPT and Claude do help.
- Find the competitors ChatGPT recommendsA method for finding who ChatGPT recommends instead of you: which questions to ask, how many times, how to count the names and work out your share of voice.
- Does ChatGPT give everyone the same answer?No. ChatGPT writes each answer afresh and can use your memory, location, a web search and your plan's model. OpenAI's own pages, and our repeat-run numbers.
- AI visibility metricsMention rate, first-pick rate, citation rate, share of voice and sentiment: what each one counts, how to work it out, and why each needs a range.
- Get ChatGPT to recommend your businessWhat gets a business named in ChatGPT's answers: what you control, whether you can submit your business, how long changes take, and how to measure it honestly.
- Get cited by ChatGPTHow ChatGPT search finds and picks the pages it links, from OpenAI's docs, what makes a page usable as a source, and which kinds of pages our data shows cited.
- Why ChatGPT doesn't show your websiteSix reasons ChatGPT leaves a site out, in the order to check them: robots.txt, firewalls, noindex, JavaScript, Bing's index and who the pages it reads name.
- Does ChatGPT use Google or Bing?What OpenAI documents about where ChatGPT search gets results (Bing, its own crawler and index), what is only reported about Google, and what it means for you.
- Rank in Google's AI OverviewsWhat Google says you need to show up in AI Overviews, how query fan-out and nosnippet work, and what studies show: 38% of cited pages rank in the top 10.
- Rank in Google AI ModeWhat sets Google's AI Mode apart from AI Overviews: query fan-out, follow-ups, a custom Gemini and Search Console's counts, and how often the two cite one page.
- Perplexity SEOHow Perplexity finds, reads and cites pages: its two crawlers, what its docs say and leave out, the Cloudflare dispute, studies, and what our runs show.
- robots.txt for AI crawlersCopy-ready robots.txt rules that keep AI search crawlers in and training crawlers out, how each crawler reads the file, and the mistakes that lock AI out.
- What is llms.txt?llms.txt is a Markdown file that tells AI agents what a site holds. Its format, a real example, who reads it today, and what studies show about its effect.
- Firewalls that block AI crawlersCloudflare, Vercel, AWS WAF and others can turn AI crawlers away even when robots.txt lets them in. Where each setting lives and how to let crawlers through.
- Do AI crawlers render JavaScript?Google's and Bing's crawlers render JavaScript. OpenAI, Anthropic and Perplexity don't say, and Vercel's 2024 study found theirs don't. Our data on cited pages.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account