Why it matters
Robots.txt is not a security measure: it sits at a predictable public address, so anyone can read the paths it disallows, and it only asks well-behaved crawlers not to fetch them. Google's guide says it is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed if other sites link to it, and Google points to password protection for private files.
How to meet it
Put admin areas, internal tools, exports and staging behind authentication or an IP allowlist (DEV-CAN-07), or keep them off the internet; do not list their paths in robots.txt. To keep a public page out of Search, use noindex and leave it crawlable (DEV-IDX-01).
How to test it
The live robots.txt names no admin, backup, export or internal paths, and curl -I on each private path returns 401 or 403 without credentials.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents01
Claims in the knowledge base01
- D1-C514ConsistentHow Google interprets robots.txtSaid on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Consistent
- Google's documentation supports it without stating it.