Give a crawler its own robots.txt group only when needed, and make that group complete

Robots.txt groups are not additive. A crawler obeys only the most specific group that names it and ignores the * group, so a Googlebot group that blocks only /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks (Dave Smart's example at Search Central Live). One group can name several crawlers, and Google combines all groups that name the same user agent into one. In the robots.txt quiz of Google's talk, a Googlebot group with only an allow rule, or an allow rule in the * group, was the wrong answer for opening /staging/preview/ to Googlebot while keeping the rest of /staging/ closed.

Put the rules every crawler shares in the * group. Add a named group (Googlebot, Storebot-Google, an AI crawler token) only when that crawler needs different rules, and copy into it every rule from the * group that should still apply to it. To give several crawlers the same rules, list their user-agent lines on top of one group. Generate the file from one template so that a change to the shared rules updates every named group too.

Decided from one URL.

An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.

How verdicts workWhat you connect