robots.txt generator for AI crawlers
Choose which AI crawlers may read your site and copy the robots.txt lines that say so. The generator covers the 13 crawlers GEOFast's X-ray tests, in three groups, and runs in your browser: nothing you choose is sent or stored.
Allow or block, group by group.
Every crawler starts as allowed. Set a whole group, or open it to give one crawler its own rule. The lines change as you choose.
Crawlers that build search answers
They find the pages an AI answer can cite.
Choose for each of its 6 crawlers
- OAI-SearchBotChatGPT search
- Claude-SearchBotClaude search
- PerplexityBotPerplexity search
- GooglebotGoogle Search & AI Overviews
- bingbotBing, Copilot & ChatGPT search index
- ApplebotSiri & Spotlight
Crawlers that fetch for a person's request
They visit a page when someone asks an assistant about it.
Choose for each of its 3 crawlers
- ChatGPT-UserChatGPT browsing
- Claude-UserClaude browsing
- Perplexity-UserPerplexity browsing
Crawlers that train models
They collect pages that future models may learn from.
Choose for each of its 4 crawlers
- GPTBotOpenAI model training
- ClaudeBotClaude model training
- Google-ExtendedGemini training & grounding
- CCBotOpen dataset used to train many models
Your robots.txt lines
# AI search crawlers: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: bingbot User-agent: Applebot Allow: / # Fetches a person asked for: allowed User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Allow: / # Model training: allowed User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Allow: /
Add the lines to the robots.txt at the root of your host. A crawler named here stops following your User-agent: * rules, so repeat any Disallow paths it should keep. Once the file is live, check it with the robots.txt checker.
How the robots.txt generator works.
- 01
Pick a rule for each group
Allow or block the crawlers that build search answers, the ones that fetch a page a person asked about, and the ones that collect pages for training.
- 02
Set single crawlers apart
Open a group to give one crawler its own rule, such as allowing CCBot while the other training crawlers stay blocked.
- 03
Add your sitemap
Optional. A full address, such as https://example.com/sitemap.xml, becomes a Sitemap line at the end.
- 04
Copy the lines into robots.txt
Each crawler gets a User-agent line with the name it matches, and each set of crawlers one rule: Allow: / or Disallow: /. Paste them into the robots.txt at the root of your host.
- 05
Check the live file
Once it's uploaded, the robots.txt checker loads the live file and shows what each of the 13 crawlers may read.
What each group does, and what blocking it costs.
Crawlers that build search answers
OAI-SearchBot, Claude-SearchBot and PerplexityBot find pages for ChatGPT search, Claude's search and Perplexity. Googlebot crawls for Google Search, AI Overviews and AI Mode included, Bingbot for Bing and Copilot, and Applebot for Siri and Spotlight. Block one and its product can no longer crawl your pages for those answers. For Googlebot and Bingbot that means their regular search results too.[1, 2, 3, 4, 5, 6, 7]
Crawlers that fetch for a person's request
ChatGPT-User, Claude-User and Perplexity-User visit a page when someone asks an assistant about it. Anthropic says all its bots follow robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, since a person asked. So a block in this group is one that only Claude-User is documented to follow.[1, 2, 3]
Crawlers that train models
Blocking GPTBot tells OpenAI not to train on your pages and leaves ChatGPT search as it is. Blocking ClaudeBot keeps your future pages out of Anthropic's training data. Blocking CCBot stops Common Crawl, whose open archive OpenAI trained GPT-3 mostly on. Google-Extended is a token, not a crawler: blocking it takes your pages out of Gemini's training and the Gemini app's grounding, while Google Search and AI Overviews stay the same.[1, 2, 4, 8, 9]
How a crawler reads the file
The file lives at the root of a host and covers only that host. A crawler follows the group that names it and ignores the User-agent: * group, so paths you disallow there stop applying to every crawler named here. Repeat those Disallow lines under the generated groups if these crawlers should stay out of them.[10, 11]
Questions about the robots.txt generator for AI crawlers.
What does a robots.txt generator for AI crawlers do?
It writes the robots.txt lines that allow or block AI crawlers by name: a User-agent line for each crawler, then Allow: / or Disallow: /. This one covers the 13 crawlers GEOFast's X-ray tests and builds the lines in your browser.
Where do the generated lines go?
In the robots.txt at the root of your host, such as example.com/robots.txt. A subdomain like blog.example.com needs its own file. Keep the rules you already have and add these lines below them.
Do the generated lines override my existing rules?
For the crawlers they name, yes. A crawler follows the group that names it and ignores the User-agent: * group, so paths you disallow there stop applying to it. Repeat those Disallow lines under the generated groups if the crawlers should stay out of them. robots.txt for AI crawlers
Should I block AI training crawlers?
That's your choice. Blocking GPTBot, ClaudeBot and CCBot keeps your pages out of model training and leaves ChatGPT search as it is. Google-Extended is different: blocking it also takes your pages out of the Gemini app's grounding. Which AI crawlers to allow
Does a Disallow line always keep an AI crawler out?
No. robots.txt is a request. OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, since a person asked for the page. A firewall rule stops them. Firewalls and AI crawlers
How do I check the file once it's live?
Run GEOFast's robots.txt checker on your site. It loads the live file and shows what each of the 13 crawlers may read, page by page. robots.txt checker for AI crawlers
Is the robots.txt generator free?
Yes. It runs in your browser with no account, and nothing you choose is sent to GEOFast or stored.
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them the vendor's own documentation, read on October 8, 2026 for the guides that cite the same pages. Vendors change these pages, so check them before you act.
- 1OpenAI: Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- 2Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- 3Perplexity: Perplexity Crawlersdocs.perplexity.ai/docs/resources/perplexity-crawlers
- 4Google: Google's common crawlersdevelopers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- 5Google: AI Features and Your Websitedevelopers.google.com/search/docs/appearance/ai-features
- 6Microsoft: Bing Webmaster Guidelinesbing.com/webmasters/help/webmaster-guidelines-30fba23a
- 7Apple: About Applebotsupport.apple.com/en-us/119829
- 8Common Crawl: CCBotcommoncrawl.org/ccbot
- 9OpenAI: Language Models are Few-Shot Learners (Table 2.2)arxiv.org/abs/2005.14165
- 10IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
- 11Google: How Google Interprets the robots.txt Specificationdevelopers.google.com/crawling/docs/robots-txt/robots-txt-spec
Keep reading
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
- How to write robots.txt for AI crawlersCopy-ready robots.txt rules that keep AI search crawlers in and training crawlers out, how each crawler reads the file, and the mistakes that lock AI out.
- Which AI crawlers should you allow?Which AI crawlers feed ChatGPT, Claude, Perplexity and Google's answers, which only train models, and what the 1,529 of 1,605 cited sites we could read allow.
- AI visibility checkerCheck your AI visibility for free: what AI crawlers can read on your site, then who ChatGPT, Perplexity and Gemini name for three of your buyers' questions.
- AI crawler checkerTest 13 AI crawlers, from GPTBot to PerplexityBot, against your robots.txt, and fetch your homepage as 4 of them. Free, no sign-up.
- robots.txt checker for AI crawlersSee what your robots.txt tells GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 9 more AI crawlers, page by page, with the lines to fix it. Free, no sign-up.
- JavaScript rendering checkerSee how much of your pages AI crawlers miss because it only appears after JavaScript runs. GEOFast renders your key pages and compares. Free, no sign-up.
- llms.txt generatorBuild an llms.txt in the llmstxt.org format, check a pasted file line by line, or see if a site publishes one. Free, with the evidence on what the file does.
- Query fan-out tool for ChatGPT and ClaudeType a question and see the web searches ChatGPT and Claude ran to answer it, with the pages each cited. Free, five questions a day.
- AI visibility change checkerIs your AI visibility change real or noise? Enter two mention counts and get each rate's 90% range and a verdict, by the rule GEOFast's dashboard uses.
- GEO auditA free GEO audit of up to 300 pages: AI crawler access, JavaScript rendering, 21 checks and fixes ranked by impact, with code. No sign-up.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account