Skip to content
regensbot
Audit a URL freeSign in
DEV-SHP-02Should2 of 4 in SHP

Let Storebot-Google crawl product, cart and checkout pages, through robots.txt and bot protection

Level
Should

Recommended: skip it only for a reason you can name.

Documentation

Documented

At least one cited Google document states or supports it.

Area
SHPShopping data (Merchant Center, product markup and Storebot)
Covered

Automatic

Decided automatically once you connect your server logs.

Why it matters

Google Shopping crawls with its own user agent, Storebot-Google, because it needs fresh prices, availability and shipping details. It visits product, cart and checkout pages, can fill in checkout forms and records price, shipping, availability, coupons and payment methods to verify the data merchants send. robots.txt rules addressed to Storebot-Google affect all Google Shopping surfaces, and a crawler without its own group follows the * group, so a shop that disallows /cart/ and /checkout/ for every crawler also blocks these checks.

How to meet it

Give Storebot-Google its own robots.txt group that repeats the general rules but leaves cart and checkout paths open; verify it like Googlebot, as one of Google's common crawlers (DEV-SRV-01), and exempt it from rate limits, CAPTCHAs and bot challenges on product, cart and checkout flows. Block it only if the shop deliberately stays out of Google Shopping.

How to test it

Google's open-source robots.txt parser, run with the Storebot-Google token, returns allowed for a product URL, /cart/ and /checkout/; server logs show Storebot-Google requests answered with 200, not 403, 429 or challenge pages.

How Regensbot covers it

AutomaticNeedsYour server logs

Decided automatically once you connect your server logs.

regensbot logs push keeps only lines from Google's and published agents' IP ranges, so no visitor IP leaves your systems. Googlebot is verified by IP and DNS.

How verdicts workWhat you connect

Sources

Google documents02

  1. 01About the Google StoreBot crawlerGoogle Merchant Center Help (opens in a new tab)
  2. 02List of Google's common crawlersGoogle (opens in a new tab)

Claims in the knowledge base06

  1. D3-C385ConsistentShopping on Search: Beyond the blue linksSaid on stage
  2. D3-C386ConfirmedShopping on Search: Beyond the blue linksSaid on stage
  3. D3-C387ConfirmedShopping on Search: Beyond the blue linksSaid on stage
  4. D3-C388SourceShopping on Search: Beyond the blue linksGoogle's documentation
  5. D3-C389SourceShopping on Search: Beyond the blue linksGoogle's documentation
  6. D3-C390n/aShopping on Search: Beyond the blue linksThe author's interpretation

Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)

Confirmed
Google's documentation states the same thing.
Consistent
Google's documentation supports it without stating it.
Source
The claim is the cited document itself.
n/a
Nothing to verify: a quotation, framing, opinion or analysis.

Audit

Check it on your site.

One audit checks a URL against all 138 requirements and returns one finding for each, with its source. One URL decides 66; for each of the others, the finding says which source to connect.

Audit a URL freeBrowse all 138