Rely only on the fields Google supports (user-agent, allow, disallow, sitemap) and keep robots.txt under 500 KiB

Robots.txt is an IETF standard, RFC 9309, that basically all major crawlers follow, and it was kept simple on purpose. Google supports only user-agent, allow, disallow and sitemap in robots.txt; it ignores other lines, such as crawl-delay or Content-Signal, so they save no crawl budget with Google but can stay for other crawlers as long as they never sit between user-agent lines (DEV-SRV-06). Google reads only the first 500 KiB of the file and generally caches it for up to 24 hours, so a change can take a day to apply. Google refreshes its cached copy every 24 hours (its service level objective, as RFC 9309 asks), and the Request a recrawl option in Search Console's robots.txt report refreshes it sooner.

Everything is allowed by default, so use allow only to re-open a path inside a disallowed one. Cover URL patterns with the * and $ wildcards instead of listing URLs: * matches any run of characters (Disallow: /*/live/ blocks /science/live/ and /sports/live/, and Allow: /science/live/ re-opens one of them), and $ ends the match (Disallow: / with Allow: /$ leaves only the home page open to that group). The longest matching rule wins and, when rules conflict, Google uses the least restrictive one. A crawler follows only the most specific group that names it, so give each named crawler a complete group (DEV-SRV-10). Start comments with # and spell field names exactly: said at Search Central Live, robots.txt is forgiving, Google skips a line it cannot parse and still uses the rest of the file, so a misspelt rule fails silently (Google's open-source parser happens to accept some misspellings of disallow and user-agent, but not of allow, and other crawlers may be stricter). Add the Sitemap line with an absolute URL. To slow Google down in an emergency, return 503 or 429 briefly instead of inventing directives. Plan robots.txt changes a day ahead; for an urgent change, request a recrawl in the robots.txt report. Listing the sitemap in robots.txt is fine, but it makes the sitemap public; submit it only in Search Console if that matters.

Decided from one URL.

An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.

How verdicts workWhat you connect