Why it matters
The crawl rate limit (hostload) is a host-wide limit shared by all Google crawlers, including Search, Ads, Shopping and Images; it applies per host, so a CDN host, an app host, subdomains and the www host may each have their own. When connect time, time to first byte or the share of 429 and 5xx responses rises, Google lowers it and crawls less, while better performance increases the crawl budget. Google's crawl budget guide does not express this limit in requests per second: it counts the parallel connections Google holds open to the server and how long they last.
How to meet it
Cache rendered HTML at the edge, keep origin response times low for uncached pages, size servers for crawler bursts, give bots exactly what an anonymous first-time visitor gets (no personalisation, the default test variant), never content or URLs that no user can get, and do not route Googlebot to a slower backend.
How to test it
Search Console Crawl stats shows a stable average response time and healthy host status; server logs compare time to first byte for verified Googlebot with users. An abrupt step down in crawling, or fewer connections opened by Googlebot, points to a lowered capacity limit rather than lower demand.
How Regensbot covers it
Decided automatically once you connect your server logs.
regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.
Sources
Google documents03
- 01Optimize your crawl budgetGoogle (opens in a new tab)
- 02Crawling infrastructureGoogle (opens in a new tab)
- 03Minimize A/B testing impact in Google SearchGoogle Search Central (opens in a new tab)
Claims in the knowledge base13
- D1-C092ConsistentHow Google thinks about crawl budgetShown on screen
- D1-C091ConfirmedHow Google thinks about crawl budgetShown on screen
- D1-C132SourceHow Google thinks about crawl budgetGoogle's documentation
- D1-C096ConsistentHow Google thinks about crawl budgetSaid on stage
- D1-C066ConsistentHow Google thinks about crawl budgetShown on screen
- D2-C024UndocumentedHow is HTML interpretedShown on screen
- D1-C141SourceHow Google thinks about crawl budgetGoogle's documentation
- D1-C374ConsistentHow Google thinks about crawl budgetSaid on stage
- D1-C376ConfirmedHow Google thinks about crawl budgetSaid on stage
- D1-C440ConfirmedQ&ASaid on stage
- D1-C493ConsistentQ&ASaid on stage
- D1-C494ConsistentQ&ASaid on stage
- D1-C547n/aQ&AThe author's interpretation
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Consistent
- Google's documentation supports it without stating it.
- Undocumented
- Said or shown at the event, not in the documentation.
- Source
- The claim is the cited document itself.
- n/a
- Nothing to verify: a quotation, framing, opinion or analysis.