Why it matters
Google treats URLs that differ as different URLs and crawls them all, even when they show the same content. A community speaker showed five kinds of similar URLs on well-known brand sites: another protocol or host, other capitalisation, extra slashes, a space encoded as %20 in one URL and + in another, and parameters in another order. Such URLs signal duplicate content, broken links and unnecessary redirects, and anyone can link to them from other sites, so the application should be hardened against them.
How to meet it
In the application, build the expected URL for each request from the route, the page type and ID (reverse routing) and the parameters the page actually uses, in a fixed order. When the requested URL differs, answer one 301 to the expected URL, or an error page when it maps to nothing. Generate internal links with the same function so they are never variants.
How to test it
Requests for the uppercase, double-slash, reordered-parameter and unused-parameter variants of a sample URL each return one 301 to the expected URL; a crawl that normalises URLs (lowercase scheme and host, default port removed, dot segments resolved, fragment dropped) finds no two crawled URLs that collapse to the same normalised URL.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents03
- 01How to specify a canonical URL with rel="canonical" and other methodsGoogle Search Central (opens in a new tab)
- 02URL structure best practices for Google SearchGoogle Search Central (opens in a new tab)
- 03Managing crawling of faceted navigation URLsGoogle (opens in a new tab)
Claims in the knowledge base08
- D1-C397ConsistentLightning session C: CrawlingSaid on stage
- D1-C399ConsistentLightning session C: CrawlingSaid on stage
- D1-C400ConsistentLightning session C: CrawlingSaid on stage
- D1-C402UndocumentedLightning session C: CrawlingSaid on stage
- D1-C403ConsistentLightning session C: CrawlingSaid on stage
- D1-C404ConsistentLightning session C: CrawlingSaid on stage
- D1-C445ConsistentQ&ASaid on stage
- D1-C378ConsistentHow Google thinks about crawl budgetSaid on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Consistent
- Google's documentation supports it without stating it.
- Undocumented
- Said or shown at the event, not in the documentation.