Set robots.txt policy per AI crawler by its own token, and know that user-triggered fetchers ignore it

Mainstream crawlers from Google, other search engines and AI companies try to follow robots.txt, and nearly all AI crawlers use their own user agents, so a site can allow search crawling and refuse AI training per company; Google's token for that is Google-Extended (DEV-IDX-11), and said at Search Central Live, other companies have similar "extended" tokens, such as Apple's Applebot-Extended, which lets a site allow Applebot for search while opting out of Apple's AI use. User-triggered fetchers, which fetch a page because a user asked, and AI agents acting for a user generally ignore robots.txt.

Write one robots.txt group per AI crawler token you want to restrict, and keep Googlebot allowed. Avoid a default-deny file that blocks every crawler and allows only chosen ones: a Google panelist called it a bad pattern, because you do not know what you are blocking. When logs show an unknown crawler fetching a lot, look up its user agent to find the token that controls it. To restrict user-triggered fetchers or agents, use authentication or bot management, not robots.txt.

Decided from one URL.

An audit reads the response, the headers, the HTML, robots.txt, the sitemaps and the structured data, and the finding shows the evidence it saw.

How verdicts workWhat you connect