GEOFast
Sign inFree X-rayRun a free X-ray

The AI crawlers, by the company that runs them

The 13 crawlers that decide whether ChatGPT, Claude, Perplexity, Google, Bing and Apple can read your site. Each page has the crawler's user agent string, its name in robots.txt, the lines that allow and block it, and what its operator says a block does.

Check your site

Checked against each operator's own documentation on October 13, 2026.

The crawlers

Pick a crawler by who runs it.

The share beside each one is how many of the 1,529 cited sites in our study close their homepage to it in robots.txt.

OpenAI

Anthropic

Perplexity

Google

Microsoft

Apple

Common Crawl

Our data

What the sites AI cites tell these crawlers.

We read the robots.txt of the 1,605 sites our recorded AI answers cited. 4.8% block at least one training crawler; 1.5% block OAI-SearchBot, Claude-SearchBot or PerplexityBot.

50 of 73

sites that block a training crawler still let OAI-SearchBot, Claude-SearchBot and PerplexityBot in.

37.6%

of the 1,500 robots.txt files name at least one of these 13 crawlers, most of them to let it in.

Cited sites whose robots.txt closes the homepage to each crawler
CCBot4.2%
GPTBot3.1%
ClaudeBot2.6%
Google-Extended2.0%
ChatGPT-User1.3%
PerplexityBot1.2%
Perplexity-User1.0%
Claude-User1.0%
OAI-SearchBot0.9%
Claude-SearchBot0.8%
Applebot0.7%
Googlebot0.1%
Bingbot0.1%

Blue: crawlers that fetch for answers. Gray: training crawlers.

We took the 1,605 sites cited in 1,033 answers from ChatGPT, Claude, Perplexity, Gemini and Google's AI Overviews and AI Mode, recorded on October 6 and 7, 2026 for 54 questions about AI visibility tools and web analytics, and read each site's robots.txt as GEOFastBot with the X-ray's own rules on October 8, 2026. These are sites AI answers already cite, so sites that shut AI search crawlers out may be rarer here than on the web at large, and a site can serve other crawlers other rules than it served us. Shares are of the 1,529 of 1,605 sites whose robots.txt we could read, or that have none; 62 refused our request.

Check and set your own rules

Free X-ray

Find out what AI says about your site.

Paste your URL and leave with a ranked list of what to fix, with steps and code.

  • Free
  • No account