Why it matters
noindex consumes crawl budget because Google must fetch the page to see it; nofollow is a hint, and the linked URLs may still be crawled through other links or sitemaps; crawl-delay is not processed at all. Google advises against noindex for crawl budget and against using robots.txt to reallocate it temporarily.
How to meet it
To stop Google fetching URLs on your own site, disallow them in robots.txt; return 404 or 410 for removed pages; consolidate duplicates with redirects or canonicals. Keep crawl rules stable instead of switching them to steer Googlebot around.
How to test it
Server logs show few or no Googlebot fetches of parameter and action URLs; robots.txt has no crawl-delay meant for Google; Crawl stats shows fewer fetches of low-value URLs after fixes.
How Regensbot covers it
Decided automatically once you connect your server logs.
regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.
Sources
Google documents03
- 01Optimize your crawl budgetGoogle (opens in a new tab)
- 02Qualify your outbound links to GoogleGoogle Search Central (opens in a new tab)
- 03How Google interprets the robots.txt specificationGoogle (opens in a new tab)
Claims in the knowledge base06
- D1-C106ConfirmedHow Google thinks about crawl budgetShown on screen
- D1-C109SourceHow Google thinks about crawl budgetGoogle's documentation
- D1-C127SourceHow Google thinks about crawl budgetGoogle's documentation
- D1-C107ConfirmedHow Google thinks about crawl budgetShown on screen
- D1-C108ConfirmedHow Google thinks about crawl budgetShown on screen
- D1-C105ConsistentHow Google thinks about crawl budgetShown on screen
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Consistent
- Google's documentation supports it without stating it.
- Source
- The claim is the cited document itself.