Google-Extended, the robots.txt token for Gemini
Google-Extended is a robots.txt token, not a crawler: it never makes a request. It tells Google whether content it already crawls from your site may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect Google Search.
Checked against Google's own documentation on October 13, 2026.
What Google-Extended is for.
Google calls Google-Extended a standalone product token that publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.[1]
Run by
Google
Its job
Model training. It collects pages that may train AI models, or decides whether pages already crawled may be used that way.
Its name in robots.txt
Google-ExtendedDoes it follow robots.txt?
It is a robots.txt token itself. Google-Extended exists only as a name in robots.txt. Google reads it there and applies it to content its other crawlers fetch.[1]
Its IP ranges
None of its own. Google-Extended has no separate user agent and makes no requests, so there is nothing to allow or block at a firewall.[1]
Search is a separate control
Google says robots.txt directives for Googlebot are the control over how a site is crawled for Search and the AI features inside it, and points to Google-Extended only for training and grounding in its other systems.[2]
No published user agent string for Google-Extended.
Google says Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with Google's existing user agent strings, and the token is used in a control capacity.[1]
The lines that allow and block Google-Extended.
Each pair of lines is a group of its own in robots.txt. The robots.txt generator writes the whole file, and the robots.txt checker reads what yours says to Google-Extended now.
Allow Google-Extended
With no robots.txt, or one that doesn't disallow it, Google-Extended is allowed already. These lines say so by name: a crawler that finds a group with its own name follows that group, and follows the
User-agent: *group only when it finds none.[3]User-agent: Google-Extended Allow: /
Block Google-Extended
Content Google crawls from the site may no longer be used to train future Gemini models, or for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.[1]
User-agent: Google-Extended Disallow: /
How many cited sites block Google-Extended.
We read the robots.txt of the 1,605 sites our recorded AI answers cited, for the 13 crawlers the X-ray reads for.
of cited sites close their homepage to Google-Extended in robots.txt: 31 of 1,529.
of 1,500 robots.txt files name Google-Extended. 29 of those close the homepage to it and 427 leave it open.
Blue: Google-Extended. Gray: the other 12 crawlers the X-ray reads for.
Google's other crawlers.
Google gives each job its own name in robots.txt, so one can be allowed and another blocked.[1]
Questions about Google-Extended.
What is Google-Extended?
Google-Extended is a robots.txt token, not a crawler. Publishers use it to manage whether content Google crawls from their sites may be used to train future Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
Does blocking Google-Extended affect Google Search or AI Overviews?
Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews and AI Mode are part of Search, and Google names robots.txt directives for Googlebot as the control for those.
Does Google-Extended have a user agent string?
No. Google says Google-Extended doesn't have a separate HTTP request user agent string: crawling is done with Google's existing user agent strings, and the token is used in a control capacity. You won't see it in server logs.
What does blocking Google-Extended do to the Gemini app?
By Google's description of the token, it covers grounding in Gemini Apps as well as training: grounding is providing content from the Google Search index to the model at prompt time. Disallowing Google-Extended opts your content out of both uses.
How many sites block Google-Extended?
In our study, 31 of 1,529 sites (2.0%) close their homepage to Google-Extended in robots.txt. The base is the sites cited in AI answers about AI visibility tools and web analytics whose robots.txt we could read, or that have none, so other fields may differ. What the cited sites allow
Where each fact comes from.
Each numbered source is the page a fact comes from, most of them Google's own documentation, read on October 13, 2026. The answers above repeat facts marked on the page. Vendors change these pages, so check them before you act.
- 1Google: Google's common crawlersdevelopers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- 2Google: AI Features and Your Websitedevelopers.google.com/search/docs/appearance/ai-features
- 3IETF: RFC 9309: Robots Exclusion Protocolrfc-editor.org/rfc/rfc9309.html
Keep reading
- The free AI crawler checkerTest your robots.txt against every crawler here, and fetch your homepage as the ones that answer questions.
- The free robots.txt checkerSee what your robots.txt tells each AI crawler, page by page.
- The free robots.txt generatorPick the crawlers to allow and block, and copy the file.
- Which AI crawlers to allowThe crawlers by the job each does, and what the sites AI cites allow.
- robots.txt for AI crawlersHow crawlers read the file, with rules to copy.
- GooglebotCrawls for Google Search, AI Overviews and AI Mode included.
- AI crawlers listThe 13 crawlers the X-ray reads for, by the company that runs them: user agents, robots.txt names and what blocking each does.
Find out what AI says about your site.
Paste your URL and leave with a ranked list of what to fix, with steps and code.
- Free
- No account