Do not put Content-Signal or other non-standard lines between user-agent lines

Googlebot ignores an unknown line placed between two user-agent lines and merges both user agents into one group, so the rules that follow apply to both. A Google slide at Search Central Live showed Bing closing the group at that line instead, so the same file blocks one crawler and not the other. Google's robots.txt talk called this the real trouble spot of the format: while RFC 9309 was being written, opinions on whether the first crawler should inherit the rules that follow split roughly 50-50. Managed robots.txt files from some CDNs add such lines, and said at Search Central Live, CDNs sometimes update a site's robots.txt without the owner's knowledge.

Keep each group as an uninterrupted run of user-agent lines followed by its allow and disallow rules. If a CDN or plug-in adds Content-Signal or other non-standard lines, make sure they sit after a group's rules, never between user-agent lines, and review the generated file after every CDN setting change. Check the version history in Search Console's robots.txt report for changes nobody on the team made.

Decided from one URL.

An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.

How verdicts workWhat you connect