How Does This Robots.txt Tester Work?
This tool tests site crawler configurations against target paths based on the official RFC 9309 internet standard. The validation processes matches rules utilizing precise regex translations mirroring Googlebot's open-source parsing behavior.
Compared to basic online linters, this engine provides exact line-number match references, enabling fast diagnostics of conflicting Allow and Disallow rules.
The 4 Google Robots.txt Rules Explained
Rule 1 — User-Agent Hierarchy
Crawlers look for rules in a top-down hierarchy: first, they check for a block exactly matching their name. If not found, they look for parent agent blocks (e.g., Googlebot-Image falls back to Googlebot). If still not found, they check the wildcard User-agent: * section. If no wildcard section exists, access is implicitly allowed.
Rule 2 — Longest Match Wins
If multiple rules in the matching block apply to a URL path, the rule with the longest path pattern string wins. This determines the precedence order, overriding file position.
Rule 3 — Tie-Breaker: Allow Wins
If an Allow rule and a Disallow rule match the path with exactly the same pattern string length, the Allow rule takes precedence per RFC 9309 §2.2.2.
Rule 4 — HTTP Status Code Logic
Web crawler behavior adapts to server responses:
| Status | What Googlebot/AI bots do |
|---|---|
| 2xx | Parse the file normally. |
| 4xx | Assume no file exists (all URLs ALLOWED). |
| 5xx | Assume server stress (all URLs BLOCKED temporarily). |
Common Robots.txt Mistakes That Hurt SEO
Configuration mistakes can block search indexing and impact rankings. Common issues include:
- Accidentally blocking Googlebot: A Disallow rule that blocks Googlebot will drop your site from search results.
- Blocking CSS and JS assets: Crawlers render pages to assess mobile usability. Blocking asset access impacts rendering.
- Directive typos: Mistakes like "Disalow" are unrecognized and ignored by search engines.
- Rules without User-agent declarations: Directives must sit under a declared User-agent block.
Testing AI Crawlers in Robots.txt
AI search engines utilize crawlers like GPTBot and ClaudeBot. To appear in generative search recommendations and conversational answers, ensure these bots are not blocked in your robots.txt.
For sites containing proprietary datasets, you can block training bots like CCBot while allowing search-oriented bots like GPTBot.
Frequently Asked Questions
Q: Is this robots.txt tester accurate?
A: Yes. The parser implements the exact matching precedence rules outlined in Google's C++ parser and the RFC 9309 specification, evaluating user-agent fallbacks, pattern lengths, and status codes.
Q: Why does the tester show ALLOWED when robots.txt has Disallow: /?
A: This occurs if a longer or more specific Allow rule matches the path. The tester shows the evaluation chain and points to the winning line number.
Q: Does robots.txt guarantee bots won't access my pages?
A: No. robots.txt is a request, not a security firewall. Trusted search and AI bots respect it, but malicious scrapers can ignore it. Use authentication or password protection for sensitive content.
Q: Why test GPTBot in robots.txt?
A: If GPTBot is blocked, your content will not be indexed or cited in ChatGPT conversational search results.
Q: What is RFC 9309?
A: RFC 9309 is the internet standard published in 2022 that codifies robots.txt parsing rules, including longest-match wins and Allow tie-breakers.
Q: What does 'implicit allow' mean?
A: If a crawler matches no specific user-agent block and no wildcard "*" section, access is allowed by default.
Q: How do I fix a URL that is blocked?
A: The tester highlights the line causing the block. To resolve it, add an Allow rule with a longer path matching the URL, or adjust the blocking Disallow rule.
Q: Can I test my live website's robots.txt?
A: Yes. Enter your domain in the fetch input to retrieve your live file via our server proxy, bypassing CORS restrictions.
Complete Your Technical SEO Toolkit
Ensure your robots.txt is error-free, then generate a spec-compliant llms.txt file to optimize your site for AI agents.