Why it matters
Said at Search Central Live: Google's duplication talk closed with the advice not to block agents, and AI agents browsing for users run into the same bot walls that sites build against scrapers and may go elsewhere, for example to buy the product from another shop.
How to meet it
Decide per agent which ones to allow, tune bot protection to challenge abusive patterns rather than every automated browser, and if a challenge is needed send it with 503 (DEV-SRV-02).
How to test it
WAF and bot-management logs show no blocks of agents the team decided to allow.
How Regensbot covers it
Regensbot proposes a verdict with the evidence; a person on your team confirms it.
The evidence comes from your server logs once you connect them.
regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.
Sources
Google documents00
No Google document is cited. This requirement was said at Search Central Live and is not documented policy.
Claims in the knowledge base03
- D2-C397UndocumentedHandling web duplicationShown on screen
- D2-C377UndocumentedHandling web duplicationSaid on stage
- D2-C378n/aHandling web duplicationThe author's interpretation
Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)
- Undocumented
- Said or shown at the event, not in the documentation.
- n/a
- Nothing to verify: a quotation, framing, opinion or analysis.