Measure the time from publishing a URL to Googlebot's first crawl of it

Said at Search Central Live by Gary Illyes: the reliable check that new content is crawled fast enough is the time between publishing a URL and Googlebot's first fetch of it; it should stay flat or fall, and only a rising trend calls for investigation, starting with URLs Google crawls and recrawls a lot without use. How much it rises matters: for fresh news stories he called something like two hours probably not great. The Crawl stats split by purpose (discovery versus refresh) gives a rougher view; John Mueller said that if about half of a site's crawling goes to discovery, crawling of new pages is not the problem. News sites rarely need to worry about crawl budget, because Google crawls them aggressively.

Log each URL's publish time in the CMS, join it with the first verified Googlebot request in the access logs, and chart the median delay per section each week. When it rises, review the most-crawled URLs in the logs and block or remove useless ones (DEV-URL-08).

Automatisch entschieden, sobald Sie Ihre Server-Logs verbinden.

regensbot logs push behält nur Zeilen aus den IP-Bereichen, die Google und KI-Agenten veröffentlichen. So verlässt keine Besucher-IP Ihre Systeme. Googlebot wird per IP und DNS verifiziert.

So funktionieren die UrteileWas Sie verbinden