AI crawler checker
See which AI crawlers your site lets in. GEOFast checks 13 of them against your robots.txt and fetches your homepage as the 4 that answer questions for ChatGPT, Claude and Perplexity.
- Free
- No account
What the AI crawler checker checks.
- 01
Reads your robots.txt
It finds the rules each crawler follows: the group written for it, or the wildcard group when none names it.
- 02
Tests your important pages
Your homepage and up to 11 more pages from the crawl, so a rule that blocks one section shows up.
- 03
Fetches your homepage as 4 crawlers
It compares their answers with a browser's, so a firewall or bot protection that turns them away shows up even when robots.txt allows them.
- 04
Ranks what to fix
Blocked answer crawlers rank high, since they decide whether AI can cite you; blocking training crawlers is your choice.
13 AI crawlers, by what blocking them costs.
Answer engines
Their crawlers decide whether you can be cited.
- OAI-SearchBotChatGPT search
- GooglebotGoogle & AI Overviews
- bingbotBing & Copilot
- PerplexityBotPerplexity
- Claude-SearchBotClaude search
- ApplebotSiri & Spotlight
Fetching for a user
Visits when someone asks an assistant about a page.
- ChatGPT-UserChatGPT
- Claude-UserClaude
- Perplexity-UserPerplexity
Model training
Blocking them keeps your pages out of training; only Google-Extended also changes AI answers.
- GPTBotOpenAI
- ClaudeBotAnthropic
- Google-ExtendedGemini
- CCBotCommon Crawl
What each verdict means.
- Allowed
robots.txt lets the crawler read every page tested.
- Partly blocked
Some important pages are off limits to it; the report names them.
- Blocked
robots.txt keeps it out. For a training crawler that's a choice; for an answer crawler it keeps you out of that AI's answers.
- Turned away
The site refused our request as the crawler and served a browser the page: a firewall or bot protection at work.
- IP-verified
Firewalls check Googlebot and Bingbot by their IP address, so the X-ray reads their robots.txt rules without fetching as them.
Numbers from our own scans.
In our test run on 30 well-known sites on October 5, 2026, 1 blocked the AI search crawlers in robots.txt, 1 allowed them there but turned them away at its firewall, and 3 turned us away at the homepage.
Questions about the AI crawler checker.
What is an AI crawler?
A bot that fetches web pages for an AI company: to answer a question now (OAI-SearchBot, PerplexityBot, Claude-SearchBot), to read a page someone asked about (ChatGPT-User, Claude-User, Perplexity-User), or to collect training data (GPTBot, ClaudeBot, CCBot). GEOFast's checker tests 13 of them.
Does the checker count a blocked GPTBot as a problem?
No. GPTBot collects pages for OpenAI's model training, so the checker reports a block on it as your choice. A block on OAI-SearchBot is a fault: ChatGPT search finds pages through that crawler. The checker shows each one separately. GPTBotWhich AI crawlers to allow
Why are AI crawlers blocked when my robots.txt allows them?
Firewalls and bot protection can refuse crawlers that robots.txt allows. The checker fetches your homepage as AI crawlers and compares the answers with a browser's, so a block like that shows up as turned away. Firewalls and AI crawlers
Why doesn't it fetch as Googlebot and Bingbot?
Firewalls verify those two by their IP address, so a request under their names from anyone else proves nothing and can get that IP flagged. The checker reads their robots.txt rules instead.
Is the AI crawler checker free?
Yes. It's part of GEOFast's free X-ray: no account, nothing to install, and most X-rays finish in under a minute.
Keep reading
- Actions that earn AI mentionsA ranked list of what gets you named: site fixes, pages to create, answers to improve and sites to get onto, then a check of whether each one moved the answers.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
- How to get cited by ChatGPTHow ChatGPT search finds and picks the pages it links, from OpenAI's docs, what makes a page usable as a source, and which kinds of pages our data shows cited.
- Why doesn't ChatGPT show your website?Six reasons ChatGPT leaves a site out, in the order to check them: robots.txt, firewalls, noindex, JavaScript, Bing's index and who the pages it reads name.
- Does ChatGPT use Google or Bing?What OpenAI documents about where ChatGPT search gets results (Bing, its own crawler and index), what is only reported about Google, and what it means for you.
- Perplexity SEO: how Perplexity finds, reads and cites pagesHow Perplexity finds, reads and cites pages: its two crawlers, what its docs say and leave out, the Cloudflare dispute, studies, and what our runs show.
- Which AI crawlers should you allow?Which AI crawlers feed ChatGPT, Claude, Perplexity and Google's answers, which only train models, and what the 1,529 of 1,605 cited sites we could read allow.
- Is your firewall blocking AI crawlers?Cloudflare, Vercel, AWS WAF and others can turn AI crawlers away even when robots.txt lets them in. Where each setting lives and how to let crawlers through.
- AI visibility checkerCheck your AI visibility for free: what AI crawlers can read on your site, then who ChatGPT, Perplexity and Gemini name for three of your buyers' questions.
- robots.txt checker for AI crawlersSee what your robots.txt tells GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and 9 more AI crawlers, page by page, with the lines to fix it. Free, no sign-up.
- robots.txt generator for AI crawlersBuild robots.txt rules for 13 AI crawlers: allow or block search, user-request and training crawlers, add a sitemap, copy the lines. Runs in your browser.
- JavaScript rendering checkerSee how much of your pages AI crawlers miss because it only appears after JavaScript runs. GEOFast renders your key pages and compares. Free, no sign-up.
- llms.txt generatorBuild an llms.txt in the llmstxt.org format, check a pasted file line by line, or see if a site publishes one. Free, with the evidence on what the file does.
- Query fan-out tool for ChatGPT and ClaudeType a question and see the web searches ChatGPT and Claude ran to answer it, with the pages each cited. Free, five questions a day.
- AI visibility change checkerIs your AI visibility change real or noise? Enter two mention counts and get each rate's 90% range and a verdict, by the rule GEOFast's dashboard uses.
- GEO auditA free GEO audit of up to 300 pages: AI crawler access, JavaScript rendering, 21 checks and fixes ranked by impact, with code. No sign-up.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account