Skip to content
regensbot
Audit a URL freeSign in
DEV-MON-11Should11 of 11 in MON

Measure the time from publishing a URL to Googlebot's first crawl of it

Level
Should

Recommended: skip it only for a reason you can name.

Documentation

Documented

At least one cited Google document states or supports it.

Area
MONTesting and monitoring
Covered

Automatic

Decided automatically once you connect your server logs.

Why it matters

Said at Search Central Live by Gary Illyes: the reliable check that new content is crawled fast enough is the time between publishing a URL and Googlebot's first fetch of it; it should stay flat or fall, and only a rising trend calls for investigation, starting with URLs Google crawls and recrawls a lot without use. How much it rises matters: for fresh news stories he called something like two hours probably not great. The Crawl stats split by purpose (discovery versus refresh) gives a rougher view; John Mueller said that if about half of a site's crawling goes to discovery, crawling of new pages is not the problem. News sites rarely need to worry about crawl budget, because Google crawls them aggressively.

How to meet it

Log each URL's publish time in the CMS, join it with the first verified Googlebot request in the access logs, and chart the median delay per section each week. When it rises, review the most-crawled URLs in the logs and block or remove useless ones (DEV-URL-08).

How to test it

The weekly chart exists per section and alerts when the median publish-to-first-crawl delay rises for several weeks in a row.

How Regensbot covers it

AutomaticNeedsYour server logs

Decided automatically once you connect your server logs.

regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.

How verdicts workWhat you connect

Sources

Google documents02

  1. 01Crawl Stats reportGoogle Search Console Help (opens in a new tab)
  2. 02Optimize your crawl budgetGoogle (opens in a new tab)

Claims in the knowledge base07

  1. D1-C467UndocumentedQ&ASaid on stage
  2. D1-C468ConsistentQ&ASaid on stage
  3. D1-C492ConsistentQ&ASaid on stage
  4. D1-C466ConsistentQ&ASaid on stage
  5. D1-C396ConfirmedHow Google thinks about crawl budgetSaid on stage
  6. D1-C543UndocumentedQ&ASaid on stage
  7. D1-C544ConsistentQ&ASaid on stage

Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)

Confirmed
Google's documentation states the same thing.
Consistent
Google's documentation supports it without stating it.
Undocumented
Said or shown at the event, not in the documentation.

Audit

Check it on your site.

One audit checks a URL against all 138 requirements and returns one finding for each, with its source. One URL decides 66; for each of the others, the finding says which source to connect.

Audit a URL freeBrowse all 138