Why it matters
Files without an HTML head can carry robots rules only in the HTTP response. X-Robots-Tag accepts the same rules as the meta tag and, like the tag, is seen only when the URL is crawled.
How to meet it
Add X-Robots-Tag in the web server or CDN for the file types or folders that should stay out of Search (internal PDFs, exports, private images), and do not also disallow those URLs in robots.txt. Use the user-agent form (X-Robots-Tag: googlebot: noindex) only for crawler-specific rules.
How to test it
curl -I on a sample file shows the X-Robots-Tag header; URL Inspection of the file reports it excluded by noindex.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents02
- 01Robots meta tag, data-nosnippet, and X-Robots-Tag specificationsGoogle Search Central (opens in a new tab)
- 02Remove images hosted on your site from search resultsGoogle Search Central (opens in a new tab)
Claims in the knowledge base03
- D2-C069SourceControlling indexingGoogle's documentation
- D2-C529SourceUsing images to your advantage and Engaging Search users with videosGoogle's documentation
- D2-C095SourceControlling indexingGoogle's documentation
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Source
- The claim is the cited document itself.