Policy builder
robots.txt
File size—
Groups—
Rules—
100
Policy audit
Checks syntax, unsupported directives, dangerous blocks and the 500 KiB Google limit.
Single URL simulator
Enter a path to test.
Crawler coverage matrix
Rule evaluation uses the most specific matching user-agent group and longest matching Allow/Disallow path; Allow wins equal-specificity ties. Wildcards * and end anchor $ are supported.
What this tool prevents
Robots.txt is crawl control, not access control.
Do not use it to protect secrets or private admin areas.
Do not use it to protect secrets or private admin areas.
noindex is different.
A URL blocked from crawling can still appear in search results if other pages link to it.
A URL blocked from crawling can still appear in search results if other pages link to it.
Crawl-delay is not portable.
Google ignores it. Use crawler-specific controls only when you know the target crawler supports them.
Google ignores it. Use crawler-specific controls only when you know the target crawler supports them.
Publication checklist
Publish exactly at /robots.txt on the protocol/host/port it controls. Keep UTF-8 plain text. Test important templates, CSS/JS paths, faceted navigation, staging rules and sitemap URLs before deployment.