Why it matters
Google-Extended is a robots.txt control token, not a crawler: it decides whether crawled content may be used to train future Gemini models and to ground Gemini Apps and Vertex AI, and it does not affect inclusion or ranking in Google Search. The Search generative AI setting does not affect training, and user-triggered fetchers, which fetch a page because a user asked, generally ignore robots.txt.
How to meet it
To opt out, add a User-agent: Google-Extended group with Disallow: / for the whole site, or Disallow lines for specific directories, to robots.txt; never block Googlebot for this purpose. Google's robots.txt talk confirmed that Google respects that policy for Gemini model training. Said at Search Central Live: rendering for Gemini training reuses the rendering done for Search. Author's view: so no separate rendering work is needed for Gemini.
How to test it
Google's open-source robots.txt parser (github.com/google/robotstxt), run with the Google-Extended token, returns disallowed for the intended paths, and run with the Googlebot token returns allowed for the same URLs; URL Inspection shows them as Crawl allowed.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents03
- 01List of Google's common crawlersGoogle (opens in a new tab)
- 02List of Google user-triggered fetchersGoogle (opens in a new tab)
- 03Search generative AI controlGoogle Search Console Help (opens in a new tab)
Claims in the knowledge base08
- D1-C086SourceHow Google interprets robots.txtGoogle's documentation
- D1-C032SourceWhat's new in the world of SearchGoogle's documentation
- D2-C121SourceLightning session D: Rendering and JavaScriptGoogle's documentation
- D2-C122SourceLightning session D: Rendering and JavaScriptGoogle's documentation
- D2-C118UndocumentedLightning session D: Rendering and JavaScriptSaid on stage
- D2-C119n/aLightning session D: Rendering and JavaScriptThe author's interpretation
- D1-C485ConfirmedQ&ASaid on stage
- D1-C522ConsistentHow Google interprets robots.txtSaid on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Consistent
- Google's documentation supports it without stating it.
- Undocumented
- Said or shown at the event, not in the documentation.
- Source
- The claim is the cited document itself.
- n/a
- Nothing to verify: a quotation, framing, opinion or analysis.