Keep faceted navigation out of the crawl with robots.txt rules or URL fragments

Facets create near-infinite URL spaces (the same filters in another order, conflicting or excessive combinations, facets on paginated or search pages), one of the things named as burning crawl budget. Google's faceted navigation guide prefers prevention: disallow filter URLs in robots.txt or use URL fragments, which Google generally does not crawl; rel=canonical and nofollow are weaker, slower options.

Decide per facet whether it deserves an indexable landing page, for example a main category or brand with real search demand, and give those a clean URL, unique content and <a href> links. Put every other filter, sort and view parameter behind a robots.txt Disallow pattern or in the # fragment, and do not link filter combinations with crawlable hrefs. Also disallow internal search results and action URLs (cart, wishlist, compare). Audit CMS plug-ins for generated URL spaces: Google's crawling team said many crawling complaints it receives turn out to come from a plug-in, such as a calendar plug-in that adds calendar parameters to the URLs of every post, page, category and tag page, and Google crawls such URLs until its systems learn from large samples that they are useless. A Google panelist added that useless plug-in parameter URLs can also be handled at the web server, for example with a rewrite rule or a custom module, so that crawl budget goes to the URLs that matter.

Regensbot schlägt mit den Belegen ein Ergebnis vor; eine Person aus Ihrem Team bestätigt es.

Die Belege stammen aus einem Crawl Ihrer Website. Wo Inhalte zu bewerten sind, schlägt Claude als Assistenz-Prüfer das Ergebnis vor und lässt eine Seite nie selbst durchfallen.

Der Crawl ist rücksichtsvoll: Sitemap zuerst, eine Anfrage pro Sekunde und Host. Ein gefundenes Problem ist eindeutig, und „nichts gefunden“ besteht nur nach einem vollständigen Crawl.

So funktionieren die UrteileWas Sie verbinden