🤖 robots.txt Viewer
Fetch and parse a site's robots.txt with security analysis — disallowed paths often reveal sensitive admin directories that attackers map first.
📚 robots.txt — security implications
robots.txt instructs search engine crawlers which paths to avoid. But it's a public file — anyone can read it, including attackers.
The security irony: Disallowing /admin/ or /internal/api/ doesn't block humans — it just tells crawlers to skip those paths. Security researchers and attackers check robots.txt to discover interesting directories the site owner might have preferred to keep quiet.
Best practice: Don't rely on robots.txt for security. Sensitive paths must be protected with proper authentication. Think of robots.txt as a signpost — if the path needs hiding, hide it properly, not just from crawlers.