robots.txt checker for AI crawlers
Find out what your robots.txt tells 13 AI crawlers, from the rules written for each one to the wildcard rules they fall back on, and which of your important pages each may read.
- Free
- No account
What the robots.txt checker for AI crawlers checks.
- 01
Loads /robots.txt
A file that errors counts as blocking everything, as it does for Google. No file at all means every crawler may read every page.
- 02
Finds each crawler's rules
A crawler follows the group that names it and ignores the wildcard group; without its own group, the wildcard rules apply.
- 03
Tests your important pages
Your homepage and up to 11 more pages, against each crawler's Allow and Disallow lines.
- 04
Writes the fix
When an answer crawler is blocked, the report gives the robots.txt lines that let it back in.
What each verdict means.
- Allowed
The crawler may read every page tested.
- Partly blocked
Some pages are off limits to it; the report lists examples.
- Blocked
Every page tested is off limits to it, or your robots.txt errors.
- Unknown
Your site turned us away, so we couldn't read its robots.txt.
Numbers from our own scans.
In our test run on 30 well-known sites on October 5, 2026, 1 blocked the AI search crawlers in robots.txt. Blocking answer crawlers keeps a site out of the answers those AIs give.
Questions about the robots.txt checker for AI crawlers.
How do I allow AI crawlers in robots.txt?
Give each crawler you want its own group, such as User-agent: OAI-SearchBot followed by Allow: /, or keep them under a wildcard group that allows your pages. A crawler follows the group that names it, so its own group wins over the wildcard. robots.txt for AI crawlersrobots.txt generator for AI crawlers
My robots.txt blocks GPTBot. Does that keep me out of ChatGPT?
No. GPTBot collects training data; ChatGPT search reads pages through OAI-SearchBot, and ChatGPT-User fetches pages people ask about. The checker reads your rules for each of the three, so you can see that the other two are still allowed. GPTBot
Does Google-Extended affect AI Overviews?
No. AI Overviews and AI Mode read pages through Googlebot. Google-Extended decides whether Gemini may train on your pages and use them to ground answers in the Gemini app, so blocking it keeps them out of those answers.
What if my site has no robots.txt?
Then every crawler may read every page, which is what AI answers need. A robots.txt that errors is worse: Google treats it as crawl nothing until it loads.
Is the robots.txt checker free?
Yes. It's part of GEOFast's free X-ray: no account, nothing to install, and most X-rays finish in under a minute.
Keep reading
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
- How to write robots.txt for AI crawlersCopy-ready robots.txt rules that keep AI search crawlers in and training crawlers out, how each crawler reads the file, and the mistakes that lock AI out.
- AI visibility checkerCheck your AI visibility for free: what AI crawlers can read on your site, then who ChatGPT, Perplexity and Gemini name for three of your buyers' questions.
- AI crawler checkerTest 13 AI crawlers, from GPTBot to PerplexityBot, against your robots.txt, and fetch your homepage as 4 of them. Free, no sign-up.
- robots.txt generator for AI crawlersBuild robots.txt rules for 13 AI crawlers: allow or block search, user-request and training crawlers, add a sitemap, copy the lines. Runs in your browser.
- JavaScript rendering checkerSee how much of your pages AI crawlers miss because it only appears after JavaScript runs. GEOFast renders your key pages and compares. Free, no sign-up.
- llms.txt generatorBuild an llms.txt in the llmstxt.org format, check a pasted file line by line, or see if a site publishes one. Free, with the evidence on what the file does.
- Query fan-out tool for ChatGPT and ClaudeType a question and see the web searches ChatGPT and Claude ran to answer it, with the pages each cited. Free, five questions a day.
- AI visibility change checkerIs your AI visibility change real or noise? Enter two mention counts and get each rate's 90% range and a verdict, by the rule GEOFast's dashboard uses.
- GEO auditA free GEO audit of up to 300 pages: AI crawler access, JavaScript rendering, 21 checks and fixes ranked by impact, with code. No sign-up.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account