Robots.txt Tester
LiveCheck if AI bots are blocked
Test your robots.txt against GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and 12 more AI crawlers. See exactly who can crawl what — free.
What is Robots.txt Tester?
Your robots.txt is the gatekeeper between your content and the AI crawlers that feed ChatGPT, Claude, and Perplexity. A single misplaced Disallow rule can quietly erase you from AI answers. Our robots.txt tester fetches your live file, parses it with Google's spec, and reports exactly which AI bots are allowed or blocked — with line-by-line explanations.
How it works
Enter your domain
We fetch https://yourdomain.com/robots.txt in real time. If none exists we tell you what that means for AI crawlers.
Parse against the spec
Our parser follows RFC 9309 (the robots.txt standard) and Google's longest-match rules so the result matches real crawler behavior.
See per-bot results
Get a color-coded grid of every AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and more) with the exact rule that applies.
Key benefits
- Tests 17+ AI crawlers in one shot
- RFC 9309 + Google longest-match compliant parser
- Line-by-line explanation of which rule applies to each bot
- Detects accidental Disallow: / that blocks everyone
- Suggests the exact fix to allow specific bots safely
Robots.txt Tester FAQ
The most common cause is a blanket Disallow: / rule, or a CMS default that blocks unknown bots. Some hosting providers (WP Engine, Kinsta) also block AI crawlers at the edge. We show you the exact rule responsible and the minimal change to allow GPTBot while keeping bad bots out.
Run the full Three Sights audit
Free tools are spot checks. The seosights platform runs all checks continuously, alerts you when something changes, and ships a 90-day auto-executed roadmap.
Start 14-day free trialNo credit card required