Why it matters
robots.txt does not control indexing: a disallowed URL can still appear in results without its content when other pages link to it, and Google never sees a noindex on a page it may not fetch. Google said it does not treat a disallow as noindex because important sites sometimes disallow their most important pages by mistake, and robots meta rules only work if robots.txt lets Google fetch the page.
How to meet it
For admin, thin, temporary or internal-search pages that may be crawled but should not be indexed, send <meta name="robots" content="noindex"> or X-Robots-Tag: noindex in the server response and do not disallow the URL. Use robots.txt only for URLs that should never be fetched. Use name="googlebot" only for Google-specific rules. To de-index URLs that are already disallowed, add the noindex first (or in the same release), then remove the disallow so Google can see it.
How to test it
For each noindexed URL: URL Inspection shows Crawl allowed: Yes, curl shows the noindex, and URL Inspection and the Page indexing report list it as excluded by the noindex tag.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents03
- 01Block Search indexing with noindexGoogle Search Central (opens in a new tab)
- 02Robots meta tag, data-nosnippet, and X-Robots-Tag specificationsGoogle Search Central (opens in a new tab)
- 03Introduction to robots.txtGoogle Search Central (opens in a new tab)
Claims in the knowledge base12
- D2-C022SourceWelcome to indexing day!Google's documentation
- D2-C069SourceControlling indexingGoogle's documentation
- D2-C021ConfirmedWelcome to indexing day!Shown on screen
- D2-C020UndocumentedWelcome to indexing day!Shown on screen
- D2-C061ConfirmedControlling indexingShown on screen
- D1-C110n/aHow Google thinks about crawl budgetThe author's interpretation
- D2-C023n/aWelcome to indexing day!The author's interpretation
- D2-C850ConfirmedControlling indexingSaid on stage
- D2-C851ConfirmedControlling indexingSaid on stage
- D2-C846ConsistentWelcome to indexing day!Said on stage
- D2-C844UndocumentedWelcome to indexing day!Said on stage
- D2-C845UndocumentedWelcome to indexing day!Said on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Consistent
- Google's documentation supports it without stating it.
- Undocumented
- Said or shown at the event, not in the documentation.
- Source
- The claim is the cited document itself.
- n/a
- Nothing to verify: a quotation, framing, opinion or analysis.