Give a crawler its own robots.txt group only when needed, and make that group complete

Robots.txt groups are not additive. A crawler obeys only the most specific group that names it and ignores the * group, so a Googlebot group that blocks only /dogs/ leaves Googlebot free to crawl the /goats/ and /cows/ that the * group blocks (Dave Smart's example at Search Central Live). One group can name several crawlers, and Google combines all groups that name the same user agent into one. In the robots.txt quiz of Google's talk, a Googlebot group with only an allow rule, or an allow rule in the * group, was the wrong answer for opening /staging/preview/ to Googlebot while keeping the rest of /staging/ closed.

Put the rules every crawler shares in the * group. Add a named group (Googlebot, Storebot-Google, an AI crawler token) only when that crawler needs different rules, and copy into it every rule from the * group that should still apply to it. To give several crawlers the same rules, list their user-agent lines on top of one group. Generate the file from one template so that a change to the shared rules updates every named group too.

Entschieden aus einer URL.

Ein Audit liest die Antwort, die Header, das HTML, robots.txt, die Sitemaps und die Structured Data, und der Befund zeigt, was es gesehen hat.

So funktionieren die UrteileWas Sie verbinden