Why it matters
If robots.txt answers with a 5xx error, Google stops crawling the site for the first 12 hours and then relies on a cached copy for up to 30 days. robots.txt rules apply per host, so the file on www.example.com does nothing for api.example.com or a CDN host name, and the file is read only at the root of a host: a robots.txt in a subdirectory is never used.
How to meet it
Serve the file at the root path /robots.txt on every host name that serves pages or rendering resources: www, the bare domain if it answers, API, static and CDN hosts, every country subdomain. Answer 200 with the rules, or 404 if the host has no rules (Google then crawls everything). Exclude robots.txt from maintenance modes, WAF challenges and authentication, and never let an API or CDN host fall back to a blanket Disallow or an error page.
How to test it
curl -I https://<each-host>/robots.txt returns 200 or 404, never 5xx, 401/403 or a redirect to a login page. Search Console's robots.txt report (Settings) shows each host's file as fetched.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents02
- 01How Google interprets the robots.txt specificationGoogle (opens in a new tab)
- 02Temporarily pause or disable a websiteGoogle Search Central (opens in a new tab)
Claims in the knowledge base04
- D1-C074SourceHow crawling errors affect SearchGoogle's documentation
- D2-C209ConfirmedLightning session D: Rendering and JavaScriptShown on screen
- D2-C210n/aLightning session D: Rendering and JavaScriptThe author's interpretation
- D1-C515ConfirmedHow Google interprets robots.txtSaid on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Source
- The claim is the cited document itself.
- n/a
- Nothing to verify: a quotation, framing, opinion or analysis.