GEOFast
Free X-rayRun a free X-ray

GEOFastBot

Last updated

GEOFastBot is the crawler behind GEOFast's free X-ray. It visits a site when someone asks GEOFast to check it, reads its public pages the way AI crawlers do, and reports what they can see. It names itself in every request:

Mozilla/5.0 (compatible; GEOFastBot/1.0; +https://trygeo.fast/bot)

What it fetches

  • Your robots.txt, your sitemaps and your llms.txt, if there is one.
  • Your homepage and up to 299 more pages, the ones that matter most first: pages your homepage links to and pages in your sitemap.
  • At most 8 requests at a time, and it stops crawling after 45 seconds.

The other requests in a scan

A scan also tests whether AI crawlers can reach your site. These requests carry the user agent of the crawler or browser under test, not GEOFastBot's:

  • OAI-SearchBot (OpenAI, ChatGPT search): your homepage, once.
  • ChatGPT-User (OpenAI, ChatGPT browsing): your homepage, once.
  • Claude-SearchBot (Anthropic, Claude search): your homepage, once.
  • PerplexityBot (Perplexity, Perplexity search): your homepage, once.
  • A regular browser: your homepage, once, to compare with the crawlers' answers.
  • A headless browser: your homepage and up to 3 more pages, rendered to see what needs JavaScript.

How it treats your site

  • It follows your robots.txt rules for GEOFastBot on every page except your homepage, which it reads to report what your site shows.
  • It stops the scan for good when your site answers 429 (too many requests) or shows a bot challenge.
  • Another check of the same site within 24 hours reuses the first scan instead of crawling again.

How to block it

Add these lines to your robots.txt:

User-agent: GEOFastBot
Disallow: /

GEOFastBot then reads only what a report needs to start: your homepage, robots.txt, sitemaps and llms.txt. Blocking its user agent at your firewall stops those requests too.

Why it exists

AI assistants can only recommend what their crawlers can read. GEOFast shows a site's owner what those crawlers see and what to fix; the AI crawler checker runs the same scan.