Never use robots.txt to hide private paths; protect them with authentication

Robots.txt is not a security measure: it sits at a predictable public address, so anyone can read the paths it disallows, and it only asks well-behaved crawlers not to fetch them. Google's guide says it is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed if other sites link to it, and Google points to password protection for private files.

Put admin areas, internal tools, exports and staging behind authentication or an IP allowlist (DEV-CAN-07), or keep them off the internet; do not list their paths in robots.txt. To keep a public page out of Search, use noindex and leave it crawlable (DEV-IDX-01).

Entschieden aus einer URL.

Ein Audit liest die Antwort, die Header, das HTML, robots.txt, die Sitemaps und die Structured Data, und der Befund zeigt, was es gesehen hat.

So funktionieren die UrteileWas Sie verbinden