Let verified Google crawlers through firewalls, CDNs and bot protection

Google does not know that a firewall or bot-protection rule is blocking it; it only sees failing requests. Timeouts, connection resets and DNS errors are treated like 5xx errors: crawling slows down at once, and indexed URLs that stay unreachable drop out of the index within days.

Allowlist Google's crawlers in WAF, CDN and bot-management rules by the IP ranges Google publishes for its common and special-case crawlers (common-crawlers.json, special-crawlers.json), or by reverse DNS to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com hosts; never by user-agent string alone (it is trivial to fake), and never by a whole googleusercontent.com host, where Google Cloud customers' machines resolve too. Do not rate-limit or challenge verified Googlebot with CAPTCHAs or JavaScript puzzles, which it cannot solve. Make sure DNS resolves reliably for every host that serves pages, scripts, styles or APIs. Review the rules after every CDN or security-vendor change, including rules written for HTTP/3 traffic or AI scrapers. When Search Console reports DNS or network errors, first ask the hosting provider, CDN or DNS provider what changed and check for recently added firewall rules: said at Search Central Live, most network errors happen between the origin and Google's data centres, where neither Google nor the site owner can see them.

Regensbot proposes a verdict with the evidence; a person on your team confirms it.

The evidence comes from your server logs once you connect them.

regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.

How verdicts workWhat you connect