Why it matters
4xx codes other than 429 do not slow crawling, URLs that return them are not indexed, and a 404 is a strong signal not to crawl the URL again. A soft 404 is kept out of the index but keeps being crawled, wasting crawl budget, and soft 404 pages can be clustered together as duplicates.
How to meet it
Answer unknown paths, deleted items without a close replacement and invalid parameters with 404 or 410 and a helpful error page (search box, main categories). Redirect only when a genuinely equivalent replacement exists, and never send every missing URL to the homepage.
How to test it
curl -I https://www.example.com/this-does-not-exist-123 returns 404; the Page indexing report's "Soft 404" count stays near zero.
How Regensbot covers it
Decided from one URL.
An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.
Sources
Google documents03
- 01How HTTP status codes affect Google's crawlersGoogle (opens in a new tab)
- 02Optimize your crawl budgetGoogle (opens in a new tab)
- 03Troubleshoot Google Search crawling errorsGoogle Search Central (opens in a new tab)
Claims in the knowledge base07
- D1-C072SourceHow crawling errors affect SearchGoogle's documentation
- D1-C073SourceHow crawling errors affect SearchGoogle's documentation
- D1-C109SourceHow Google thinks about crawl budgetGoogle's documentation
- D2-C334ConfirmedUnderstanding what's on a pageSaid on stage
- D2-C367ConsistentHandling web duplicationSaid on stage
- D1-C351ConfirmedHow crawling errors affect SearchSaid on stage
- D1-C385ConsistentHow Google thinks about crawl budgetSaid on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Consistent
- Google's documentation supports it without stating it.
- Source
- The claim is the cited document itself.