Why it matters
5xx and 429 responses make Google slow down temporarily and keep already indexed URLs for a while, but URLs that keep failing are eventually dropped. A 200 "we'll be back soon" page becomes a soft 404 or replaces your content in the index, and a 404 removes the URLs. Google's crawl rate guide says crawling picks up again automatically once the errors drop. Said at Search Central Live (October 2026): Google lowers a site's crawl capacity within about four hours of a burst of 500 errors, while increases take one to three weeks, so a botched maintenance window can cost weeks of crawling.
How to meet it
During maintenance return 503 for every page (Retry-After is optional standard HTTP), keep robots.txt answering normally, and limit the 503 period to one or two days. For a longer pause keep the site online with reduced functionality, for example a disabled cart and an information banner, and update Product markup to the real availability. Use 503 or 429 only for genuine outages or overload, never as a permanent throttle.
How to test it
In a maintenance drill, curl -I on any page returns 503 while curl -I on /robots.txt returns 200. Afterwards, the 5xx share in Crawl stats falls back to zero.
How Regensbot covers it
Regensbot proposes a verdict with the evidence; a person on your team confirms it.
The evidence comes from your verified site: its pages, its profile and the files you upload.
You verify the site by a DNS TXT record, a meta tag or a file. Regensbot then keeps a checklist item for this requirement, with a state, a guide, an evidence upload and reminders.
Sources
Google documents04
- 01Temporarily pause or disable a websiteGoogle Search Central (opens in a new tab)
- 02How HTTP status codes affect Google's crawlersGoogle (opens in a new tab)
- 03Debug network and DNS errors for Google's crawlersGoogle (opens in a new tab)
- 04Reduce the Google crawl rateGoogle (opens in a new tab)
Claims in the knowledge base11
- D1-C071SourceHow crawling errors affect SearchGoogle's documentation
- D1-C126SourceHow crawling errors affect SearchGoogle's documentation
- D2-C012SourceWelcome to indexing day!Google's documentation
- D1-C078n/aHow crawling errors affect SearchThe author's interpretation
- D2-C342n/aUnderstanding what's on a pageThe author's interpretation
- D3-C618ConsistentHow long does it take to..?Said on stage
- D3-C619ConsistentHow long does it take to..?Said on stage
- D3-C668SourceHow long does it take to..?Google's documentation
- D3-C621n/aHow long does it take to..?The author's interpretation
- D1-C353ConfirmedHow crawling errors affect SearchSaid on stage
- D1-C354ConfirmedHow crawling errors affect SearchSaid on stage
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Confirmed
- Google's documentation states the same thing.
- Consistent
- Google's documentation supports it without stating it.
- Source
- The claim is the cited document itself.
- n/a
- Nothing to verify: a quotation, framing, opinion or analysis.