Monitor Crawl stats and server logs for verified Googlebot traffic

Firewalls, DNS problems and robots.txt errors are blind spots, because Google only sees failing requests, and response times and 429 or 5xx rates drive how much Google crawls. Logs show what Googlebot actually fetches, which matters on sites with filter parameters or complicated URLs, where crawlers can easily wander off into URLs that make no sense for the site, and client-side analytics may not see Googlebot at all, because Google's renderer may skip reporting and error requests.

Keep access logs with status code, response time and user agent (logs alone do not show where Google's crawl limit lies, so read them with Crawl stats: total crawling, average response time, the response and file-type breakdowns and the discovery versus refresh split), verify Googlebot against Google's published crawler IP ranges or by reverse DNS to crawl-*.googlebot.com when analysing them, and alert on spikes of 5xx or 429 responses, robots.txt fetch failures and DNS errors. Review the Crawl stats host status regularly.

Regensbot proposes a verdict with the evidence; a person on your team confirms it.

The evidence comes from your server logs once you connect them.

regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.

How verdicts workWhat you connect