Keep faceted navigation out of the crawl with robots.txt rules or URL fragments

Facets create near-infinite URL spaces (the same filters in another order, conflicting or excessive combinations, facets on paginated or search pages), one of the things named as burning crawl budget. Google's faceted navigation guide prefers prevention: disallow filter URLs in robots.txt or use URL fragments, which Google generally does not crawl; rel=canonical and nofollow are weaker, slower options.

Decide per facet whether it deserves an indexable landing page, for example a main category or brand with real search demand, and give those a clean URL, unique content and <a href> links. Put every other filter, sort and view parameter behind a robots.txt Disallow pattern or in the # fragment, and do not link filter combinations with crawlable hrefs. Also disallow internal search results and action URLs (cart, wishlist, compare). Audit CMS plug-ins for generated URL spaces: Google's crawling team said many crawling complaints it receives turn out to come from a plug-in, such as a calendar plug-in that adds calendar parameters to the URLs of every post, page, category and tag page, and Google crawls such URLs until its systems learn from large samples that they are useless. A Google panelist added that useless plug-in parameter URLs can also be handled at the web server, for example with a rewrite rule or a custom module, so that crawl budget goes to the URLs that matter.

Regensbot proposes a verdict with the evidence; a person on your team confirms it.

The evidence comes from a crawl of your site. Where content needs judging, Claude proposes the verdict as an assistant judge and never fails a page on its own.

The crawl is polite: sitemap first, one request per second per host. A problem it finds is definite, and “none found” passes only on a complete crawl.

How verdicts workWhat you connect