Keep robots.txt reachable on every host: 200 with rules, or 404 if there are none

If robots.txt answers with a 5xx error, Google stops crawling the site for the first 12 hours and then relies on a cached copy for up to 30 days. robots.txt rules apply per host, so the file on www.example.com does nothing for api.example.com or a CDN host name, and the file is read only at the root of a host: a robots.txt in a subdirectory is never used.

Serve the file at the root path /robots.txt on every host name that serves pages or rendering resources: www, the bare domain if it answers, API, static and CDN hosts, every country subdomain. Answer 200 with the rules, or 404 if the host has no rules (Google then crawls everything). Exclude robots.txt from maintenance modes, WAF challenges and authentication, and never let an API or CDN host fall back to a blanket Disallow or an error page.

Entschieden aus einer URL.

Ein Audit liest die Antwort, die Header, das HTML, robots.txt, die Sitemaps und die Structured Data, und der Befund zeigt, was es gesehen hat.

So funktionieren die UrteileWas Sie verbinden