Never route a crawlable URL's redirects through a URL that robots.txt blocks

Said at Search Central Live by Dave Smart: robots.txt is checked for every URL in a redirect chain and crawling stops at the first blocked one. His example was a shop that redirected every page through a disallowed /cart/ URL with JavaScript to set the local currency and back, so its pages were reported as blocked by robots.txt. It applies to every kind of redirect, for example through an external authorisation service whose own robots.txt blocks crawlers, through an old intermediate URL that was disallowed years after the content moved past it, or a redirect served only to Googlebot's user agent. Search Console shows the block on the first URL of the chain, which is not itself disallowed; Google's URL Inspection help confirms that the live test follows redirects without saying so or showing the final URL it tested.

Set currency, language, consent or session state on the page itself or with a cookie, not with a redirect hop through cart, checkout, login or tracking URLs. Point every redirect straight at its final URL (DEV-CAN-01) and re-check the redirect map whenever robots.txt gains a Disallow rule, so a historical intermediate URL cannot block the target. Do not send crawlers through third-party hosts in a redirect chain, and never redirect Googlebot differently from users (DEV-SPM-03).

Decided from one URL.

An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.

How verdicts workWhat you connect