Do not answer crawlers with 401, 402 or 403 on pages that should be in Search

Google does not index URLs that return a 4xx status (except 429) and removes already indexed ones. Said at Search Central Live by Gary Illyes: Google has recently seen more 403 responses to its crawlers (described on stage as "authentication required"), treats them as client errors like 404 or 410 and drops those pages from Search and its AI features; more recently still it has seen an uptick in 402 Payment Required responses, which it treats as a 404 because Googlebot cannot pay. A 403 tells the crawler it may not see the content, so there is nothing to index.

Check that login walls, paywalls, geo-blocks and bot-management rules never return 401, 402 or 403 to verified Google crawlers on public pages. For paywalled articles serve the content with the paywalled-content markup instead of an error (DEV-SDA-13). Keep 401 and 403 for pages that must stay out of Search, such as account areas.

Decided automatically once you connect Google: Search Console, and Merchant Center or BigQuery where they apply.

One consent screen grants read-only Search Console access. Merchant Center and BigQuery are requested separately, only where they apply, and Regensbot only reads from them. URL Inspection samples and Search Analytics add Google's own view of your pages.

How verdicts workWhat you connect