Skip to content
regensbot
Audit a URL freeSign in
DEV-REN-04Avoid4 of 10 in REN

Do not block JavaScript, CSS or API endpoints that pages need for rendering in robots.txt

Level
Avoid

Must not: the site is not ready to launch while it does this.

Documentation

Documented

At least one cited Google document states or supports it.

Area
RENRendering and JavaScript
Covered

Automatic

Decided automatically once Regensbot renders your pages.

Why it matters

Google's renderer fetches scripts, styles and API requests through Googlebot and obeys robots.txt: if it cannot fetch a resource, it cannot run it. Blocking a script folder or a generic /api/ folder leaves blank modules or an empty page, which can be treated as a soft 404 although users see a full page.

How to meet it

Allow all JS, CSS and font paths. If an API folder must stay disallowed, carve out the endpoints rendering needs (Disallow: /api/ plus Allow: /api/products/), since the longer Allow rule wins. Check the robots.txt of every host the page loads data from, and re-test a rendered page after every robots.txt change touching script or API paths.

How to test it

URL Inspection live test, page resources: nothing needed for content is "Blocked by robots.txt". The DevTools Network panel (Fetch/XHR) lists each API host, and each host's robots.txt allows those paths.

How Regensbot covers it

AutomaticNeedsRendering in headless Chromium

Decided automatically once Regensbot renders your pages.

Headless Chromium renders the page on an isolated render service, and the raw and the rendered HTML run through the same parsers.

How verdicts workWhat you connect

Sources

Google documents04

  1. 01Introduction to robots.txtGoogle Search Central (opens in a new tab)
  2. 02Understand the JavaScript SEO basicsGoogle Search Central (opens in a new tab)
  3. 03Crawling December: The how and why of Googlebot crawlingSearch Central blog (3 December 2024) (opens in a new tab)
  4. 04Inside Googlebot: demystifying crawling, fetching, and the bytes we processSearch Central blog (31 March 2026) (opens in a new tab)

Claims in the knowledge base07

  1. D2-C204ConfirmedLightning session D: Rendering and JavaScriptShown on screen
  2. D2-C206SourceLightning session D: Rendering and JavaScriptGoogle's documentation
  3. D2-C297SourceWhat is Google friendly JavaScriptGoogle's documentation
  4. D2-C207ConsistentLightning session D: Rendering and JavaScriptShown on screen
  5. D2-C265ConsistentWhat is Google friendly JavaScriptShown on screen
  6. D2-C296ConfirmedWhat is Google friendly JavaScriptSaid on stage
  7. D2-C213ConfirmedLightning session D: Rendering and JavaScriptSaid on stage

Claim IDs refer to the Search Central Live 2026 knowledge base. The knowledge base (opens in a new tab)

Confirmed
Google's documentation states the same thing.
Consistent
Google's documentation supports it without stating it.
Source
The claim is the cited document itself.

Audit

Check it on your site.

One audit checks a URL against all 138 requirements and returns one finding for each, with its source. One URL decides 66; for each of the others, the finding says which source to connect.

Audit a URL freeBrowse all 138