GEOFast
Sign inFree X-rayRun a free X-ray

Googlebot, the crawler behind Google Search and its AI features

Googlebot is the crawler Google uses to build its search index. Google says robots.txt rules for Googlebot are the control over how a site is crawled for Search, and that a page must be eligible to show in Search with a snippet to be a supporting link in AI Overviews or AI Mode.

Check your site for Googlebot

Checked against Google's own documentation on October 13, 2026.

The crawler

What Googlebot is for.

Google says its common crawlers find information for building its search indexes, and that crawling preferences addressed to Googlebot affect Google Search, Discover and all Search features included, and products such as Google Images, Video and News.[1]

  • Run by

    Google

  • Its job

    Search index. It crawls on its own schedule, to find pages for search results.

  • Its name in robots.txt

    Googlebot

  • Does it follow robots.txt?

    Yes, by its operator's account. Google says its common crawlers always obey robots.txt rules when crawling automatically.[1]

  • Its IP ranges

    Google publishes the IP ranges of its common crawlers, Googlebot among them, as a JSON file. Their reverse DNS names match crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.[1, 2] developers.google.com/static/crawling/ipranges/common-crawlers.json

  • AI Overviews have no token of their own

    Google says AI is built into Search, which is why robots.txt directives for Googlebot are the control over how a site is crawled for it.[3]

  • Google-Extended is a separate choice

    To limit AI training and grounding in Google's other systems, Google points to the Google-Extended token, which does not affect a site's inclusion in Google Search.[1, 3]

User agent

The Googlebot user agent strings.

W.X.Y.Z stands for the version of Chrome Googlebot runs, which rises over time. Google warns that the user agent string can be spoofed, and gives a way to verify a request.[1, 2]

  • Googlebot Smartphone

    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

  • Googlebot Desktop

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

  • Rarely, the long form

    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

  • Rarely, the short form

    Googlebot/2.1 (+http://www.google.com/bot.html)
robots.txt

The lines that allow and block Googlebot.

Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to Googlebot now.

  • Allow Googlebot

    With no robots.txt, or one that doesn't disallow it, Googlebot is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the User-agent: * group only when it finds none.[4]

    User-agent: Googlebot
Allow: /

  • Block Googlebot

    Google can no longer crawl the pages for Search. Google says a page must be indexed and eligible to be shown in Search with a snippet to appear as a supporting link in AI Overviews or AI Mode. To limit only what Search shows from a page, it names nosnippet, data-nosnippet, max-snippet and noindex.[1, 3]

    User-agent: Googlebot
Disallow: /
Our data

How many cited sites block Googlebot.

We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.

0.1%

of cited sites close their homepage to Googlebot in robots.txt: 1 of 1,529.

Cited sites whose robots.txt closes the homepage to each crawler
CCBot4.2%
GPTBot3.1%
ClaudeBot2.6%
Google-Extended2.0%
ChatGPT-User1.3%
PerplexityBot1.2%
Perplexity-User1.0%
Claude-User1.0%
OAI-SearchBot0.9%
Claude-SearchBot0.8%
Applebot0.7%
Googlebot0.1%
Bingbot0.1%

Blue: Googlebot. Gray: the other 12 crawlers the X-ray reads for.

We took the 1,605 sites cited in 1,033 answers from ChatGPT, Claude, Perplexity, Gemini and Google's AI Overviews and AI Mode, recorded on October 6 and 7, 2026 for 54 questions about AI visibility tools and web analytics, and read each site's robots.txt as GEOFastBot with the X-ray's own rules on October 8, 2026. These are sites AI answers already cite, so sites that shut AI search crawlers out may be rarer here than on the web at large, and a site can serve other crawlers other rules than it served us. Shares are of the 1,529 of 1,605 sites whose robots.txt we could read, or that have none; 62 refused our request.
Same operator

Google's other crawlers.

Google gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]

FAQ

Questions about Googlebot.

  • Does Googlebot crawl for AI Overviews?

    Yes. Google says AI is built into Search, so robots.txt directives for Googlebot are the control over how a site is crawled for it. To be a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Search with a snippet.

  • What is the Googlebot user agent string?

    Google lists several. Googlebot Desktop is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36, where W.X.Y.Z is the Chrome version in use. Googlebot Smartphone carries the same Googlebot/2.1 part in an Android Chrome string.

  • How do I verify that a request is from Googlebot?

    Google says the user agent string can be spoofed. Run a reverse DNS lookup on the IP address and check that the name is under googlebot.com, google.com or googleusercontent.com, then confirm with a forward lookup that it returns the same address. Or match the address against the IP ranges Google publishes in common-crawlers.json.

  • Can I block Google's AI without blocking Googlebot?

    Partly. Google-Extended is a robots.txt token that covers training future Gemini models and grounding in Gemini Apps and Vertex AI, and Google says it does not affect Search. AI Overviews and AI Mode are part of Search and follow Googlebot's rules, with nosnippet, data-nosnippet and max-snippet to limit what they show.

  • How many sites block Googlebot?

    In our study, 1 of 1,529 sites (0.1%) close their homepage to Googlebot in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow

Sources

Where each fact comes from.

Each numbered source is the page a fact comes from, most of them Google's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.

  1. 1Google: Google's common crawlersdevelopers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
  2. 2Google: Verify Requests from Google Crawlers and Fetchersdevelopers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
  3. 3Google: AI Features and Your Websitedevelopers.google.com/search/docs/appearance/ai-features
  4. 4IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html

Keep reading

Free X-ray

Find out what AI says about your site.

Paste your URL and leave with a ranked list of what to fix, with steps and code.

  • Free
  • No account