Let verified Google crawlers through firewalls, CDNs and bot protection

Google does not know that a firewall or bot-protection rule is blocking it; it only sees failing requests. Timeouts, connection resets and DNS errors are treated like 5xx errors: crawling slows down at once, and indexed URLs that stay unreachable drop out of the index within days.

Allowlist Google's crawlers in WAF, CDN and bot-management rules by the IP ranges Google publishes for its common and special-case crawlers (common-crawlers.json, special-crawlers.json), or by reverse DNS to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com hosts; never by user-agent string alone (it is trivial to fake), and never by a whole googleusercontent.com host, where Google Cloud customers' machines resolve too. Do not rate-limit or challenge verified Googlebot with CAPTCHAs or JavaScript puzzles, which it cannot solve. Make sure DNS resolves reliably for every host that serves pages, scripts, styles or APIs. Review the rules after every CDN or security-vendor change, including rules written for HTTP/3 traffic or AI scrapers. When Search Console reports DNS or network errors, first ask the hosting provider, CDN or DNS provider what changed and check for recently added firewall rules: said at Search Central Live, most network errors happen between the origin and Google's data centres, where neither Google nor the site owner can see them.

Regensbot schlägt mit den Belegen ein Ergebnis vor; eine Person aus Ihrem Team bestätigt es.

Die Belege stammen aus Ihren Server-Logs, sobald Sie diese verbinden.

regensbot logs push behält nur Zeilen aus den IP-Bereichen, die Google und KI-Agenten veröffentlichen. So verlässt keine Besucher-IP Ihre Systeme. Googlebot wird per IP und DNS verifiziert.

So funktionieren die UrteileWas Sie verbinden