How to write robots.txt for AI crawlers
Give each AI crawler you care about its own group. A crawler that finds its name follows that group and ignores the wildcard one, so a dozen lines let OAI-SearchBot, Claude-SearchBot and PerplexityBot in and keep GPTBot, ClaudeBot and CCBot out.
Checked against each vendor's own docs on October 8, 2026.
Three robots.txt files, by what you want.
The file lives at the root of each host, such as example.com/robots.txt, and covers only that host: blog.example.com needs its own.[1, 2]
In AI answers, out of training
Search crawlers and user fetchers may read every page; training crawlers may read none. Google-Extended stays out of the list, since blocking it also takes you out of the Gemini app's grounding.[3] Keep any private paths as
Disallowlines in every group.# AI search, and fetches people ask for User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Model training User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot Disallow: / # Everyone else User-agent: * Allow: /
Private paths for all, training for none
Crawlers you don't name follow the
*group and skip /admin/ and /cart/. The training crawlers get a group of their own, which replaces the*group for them, so they read nothing.[1]User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot Disallow: /
Out of AI, still in Google and Bing
The AI crawlers this file names stay out if they follow robots.txt, while Googlebot and Bingbot keep your search listings. OpenAI, Perplexity, Meta and Amazon say their user fetchers may not follow it, so block those at your firewall.[4, 5, 6, 7]
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: Meta-WebIndexer User-agent: meta-externalfetcher User-agent: DuckAssistBot Disallow: /
Build your own file
The free robots.txt generator writes these lines for the crawlers you choose. Once the file is live, the robots.txt checker shows what it tells each crawler.
Look up a crawler
Each of these has a page with its user agent string, its IP ranges and what its operator says a block does: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, CCBot. The AI crawlers list has them by the company that runs each.
Seven rules for how a crawler reads the file.
Names match in any case
User-agent: gptbotandUser-agent: GPTBotname the same crawler.[1, 8]Its own group replaces the * group
A crawler follows the most specific group that names it and ignores the
*group; only a crawler with no group of its own falls back to*. SeveralUser-agentlines can share one group.[1, 8]The longest path wins
When an
Allowand aDisallowline both match a URL, the one with the longer path decides. On a tie, Google picks the less restrictive rule.[1]No file allows all; an error blocks all
A 404 means there are no rules. A 5xx error means crawl nothing: Google stops for 12 hours, then works from its last good copy for up to 30 days.[1, 8]
Changes take a day or more
Google caches the file for up to 24 hours. OpenAI says about 24 hours, Perplexity and Meta up to 24 hours, and DuckDuckGo 72 hours.[1, 4, 5, 6, 9]
Only the first 500 KiB counts
Google reads the first 500 kibibytes and ignores the rest.[1]
It's a request, not a lock
The standard says its rules aren't a form of access authorization. A crawler that ignores them can still fetch your pages; only a firewall stops it.[8]
Files that name AI crawlers mostly let them in.
We read the 1,500 robots.txt files served by the sites our recorded AI answers cited, and looked for the 13 AI crawlers the X-ray checks.
of the 1,500 files name at least one of those crawlers.
files that name GPTBot close their homepage to it; the other 446 leave it open.
files that name OAI-SearchBot, ChatGPT's search crawler, close their homepage to it.
files carry a Content-Signal line. 41 of them say ai-train=no, and none says search=no.
Seven ways a robots.txt goes wrong for AI.
Disallow: /underUser-agent: *It shuts out every crawler you don't name, AI search crawlers included. 12 of the 13 sites in our study with this line give AI crawlers groups of their own, which is how they let some back in. Without those groups, the file closes the site to every AI crawler.
Blocking the wrong OpenAI crawler
GPTBot collects training data; OAI-SearchBot and ChatGPT-User fetch what ChatGPT reads for answers. Disallowing all three to stop training stops the answers too.[4]
Expecting Google-Extended to cover AI Overviews
It doesn't. Googlebot's rules decide what AI Overviews and AI Mode may use, and the
nosnippetandmax-snippettags limit what they quote.[10, 11]A robots.txt that errors
A 5xx answer tells crawlers to crawl nothing, whatever the file says when it loads. Check what your server answers for /robots.txt, not only what the file contains.[8]
A firewall in front of the file
62 sites in our study refused our crawler's request for robots.txt itself. A firewall that refuses AI crawlers closes your pages to them whatever the file says. Check your firewall's settings.
A token with the wrong hyphen
Perplexity's crawler table writes Perplexity‑User with a non-breaking hyphen. Pasted into robots.txt, that line names no crawler. Type the tokens, or copy them from the user-agent strings.[5]
Forgetting subdomains
Each host reads only its own file, so docs.example.com and blog.example.com need theirs.[1, 2]
A Content-Signal line states a preference.
Cloudflare's Content Signals Policy adds a line that says whether your content may be used for search, as input to AI answers and for training. Cloudflare calls the signals preferences, not technical countermeasures.[12] None of the OpenAI, Anthropic, Perplexity or Google crawler pages we read mentions them.
Questions about robots.txt for AI.
Where does robots.txt go?
At the root of each host, such as example.com/robots.txt. A subdomain like blog.example.com needs its own file.
How do I allow all AI crawlers?
With no robots.txt, or one whose rules don't block them, every crawler may read every page. If your * group disallows paths AI crawlers should read, give those crawlers a group of their own.
How long until AI crawlers see a change?
Up to a day for most, longer for some. Google caches robots.txt for up to 24 hours, OpenAI says about 24 hours, Perplexity up to 24 hours, and DuckDuckGo 72 hours for DuckAssistBot.
Is User-agent case-sensitive?
No. RFC 9309 and Google both match crawler names without regard to case, so GPTBot and gptbot are the same crawler.
Can robots.txt stop ChatGPT from reading a page someone pastes?
Not for certain. OpenAI says robots.txt rules may not apply to ChatGPT-User, which fetches the pages people ask about. A firewall rule stops it. Firewalls and AI crawlers
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them the vendor's own documentation, read on October 8, 2026. The answers above repeat facts marked in the guide. Vendors change these pages, so check them before you act.
- 1Google: How Google Interprets the robots.txt Specificationdevelopers.google.com/crawling/docs/robots-txt/robots-txt-spec
- 2Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 3Google: Google's common crawlersdevelopers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- 4OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- 5Perplexity: Perplexity Crawlersdocs.perplexity.ai/docs/resources/perplexity-crawlers
- 6Meta: Meta web crawlersdevelopers.facebook.com/documentation/sharing/webmasters/web-crawlers
- 7Amazon: About Amazonbotdeveloper.amazon.com/amazonbot
- 8IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
- 9DuckDuckGo: Is DuckAssistBot related to DuckDuckGo?duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- 10Google: AI Features and Your Websitedevelopers.google.com/search/docs/appearance/ai-features
- 11Google: Robots Meta Tags Specificationsdevelopers.google.com/search/docs/crawling-indexing/robots-meta-tag
- 12Cloudflare: Giving users choice with Cloudflare's new Content Signals Policyblog.cloudflare.com/content-signals-policy/
- 13Cloudflare: robots.txt settingdevelopers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
- 14Cloudflare: Have it both ways: stay discoverable in search while disallowing AI trainingblog.cloudflare.com/accountable-mixed-use-ai-crawlers/
Keep reading
- The free robots.txt checker for AI crawlersSee what your robots.txt tells GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 9 more AI crawlers, page by page, with the lines to fix it. Free, no sign-up.
- The free robots.txt generator for AI crawlersBuild robots.txt rules for 13 AI crawlers: allow or block search, user-request and training crawlers, add a sitemap, copy the lines. Runs in your browser.
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI visibility tools, comparedEvery tool we have checked, by starting price.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
- What is GEOGEO, generative engine optimization, in plain words: how AI answers pick their sources, what research shows helps, and how to measure your AI visibility.
- AEO vs GEO vs SEOAEO vs GEO vs SEO: where each name came from, what Google and Microsoft say, the work all three share, and the four things that change with AI answers.
- Check if ChatGPT mentions your brandCheck by hand in five steps, see why one chat can't measure it (answers change, memory, location, leading questions), and where ChatGPT and Claude do help.
- Find the competitors ChatGPT recommendsA method for finding who ChatGPT recommends instead of you: which questions to ask, how many times, how to count the names and work out your share of voice.
- Does ChatGPT give everyone the same answer?No. ChatGPT writes each answer afresh and can use your memory, location, a web search and your plan's model. OpenAI's own pages, and our repeat-run numbers.
- AI visibility metricsMention rate, first-pick rate, citation rate, share of voice and sentiment: what each one counts, how to work it out, and why each needs a range.
- Get ChatGPT to recommend your businessWhat gets a business named in ChatGPT's answers: what you control, whether you can submit your business, how long changes take, and how to measure it honestly.
- Get cited by ChatGPTHow ChatGPT search finds and picks the pages it links, from OpenAI's docs, what makes a page usable as a source, and which kinds of pages our data shows cited.
- Why ChatGPT doesn't show your websiteSix reasons ChatGPT leaves a site out, in the order to check them: robots.txt, firewalls, noindex, JavaScript, Bing's index and who the pages it reads name.
- Does ChatGPT use Google or Bing?What OpenAI documents about where ChatGPT search gets results (Bing, its own crawler and index), what is only reported about Google, and what it means for you.
- Rank in Google's AI OverviewsWhat Google says you need to show up in AI Overviews, how query fan-out and nosnippet work, and what studies show: 38% of cited pages rank in the top 10.
- Rank in Google AI ModeWhat sets Google's AI Mode apart from AI Overviews: query fan-out, follow-ups, a custom Gemini and Search Console's counts, and how often the two cite one page.
- Perplexity SEOHow Perplexity finds, reads and cites pages: its two crawlers, what its docs say and leave out, the Cloudflare dispute, studies, and what our runs show.
- Which AI crawlers to allowWhich AI crawlers feed ChatGPT, Claude, Perplexity and Google's answers, which only train models, and what the 1,529 of 1,605 cited sites we could read allow.
- What is llms.txt?llms.txt is a Markdown file that tells AI agents what a site holds. Its format, a real example, who reads it today, and what studies show about its effect.
- Firewalls that block AI crawlersCloudflare, Vercel, AWS WAF and others can turn AI crawlers away even when robots.txt lets them in. Where each setting lives and how to let crawlers through.
- Do AI crawlers render JavaScript?Google's and Bing's crawlers render JavaScript. OpenAI, Anthropic and Perplexity don't say, and Vercel's 2024 study found theirs don't. Our data on cited pages.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account