Never serve bot challenges or error pages to crawlers with a 200 status

Google finds a bot-challenge page very hard to recognise as an error. Served with 200 on many URLs, the identical page can make Google cluster those URLs as duplicates and drop them, and recovery is slow because Google has little reason to recrawl duplicates. A 200 error page is also a soft 404.

If a crawler must get a challenge or block page, send it with 503 (temporary), as Google's CDN guidance recommends for bot-verification interstitials, or 429 for rate limiting; never 200. Most CDN and bot-management products let you set the status code of challenge responses. Better still, do not challenge verified Google crawlers at all (DEV-SRV-01). A CDN that throttles crawlers by injecting 429 or 503 responses on the network path shows up as a rise in HTTP errors in crawl reports.

Entschieden aus einer URL.

Ein Audit liest die Antwort, die Header, das HTML, robots.txt, die Sitemaps und die Structured Data, und der Befund zeigt, was es gesehen hat.

So funktionieren die UrteileWas Sie verbinden