Traffichood / Site diagnostics
← All SEO toolsRobots.txt and AI crawler checker
See which robots.txt rules apply to a page. Compare search and AI crawler policies, or test pasted rules before changing your live file.
The service processes the URL or content you submit. URL checks request the public page. Avoid confidential URLs and content. Downloaded reports stay on your device; contact us only when you choose to share your findings. Privacy policy
Example to try
For a local rule example, use https://example.com/private/ and paste “User-agent: *” followed by “Disallow: /private/” on the next line. This tests a restriction without changing any website.
Your diagnostic report
Your report will appear here. Basic guest checks work without an account; Google sign-in unlocks the full free report.
- 01 Enter the specific page URL whose path you want to test.
- 02 Leave the rules field empty to fetch robots.txt, or paste a proposed file without publishing it.
- 03 Read the matched rule for each crawler. Check firewall controls and actual server logs separately.
Your diagnostic report
- Passed
- 0
- To review
- 0
- Issues
- 0
What this report cannot tell you
When to use it
For website owners investigating crawl restrictions and teams reviewing search, AI search or AI training crawler policies.
How to use this check
- Enter the specific page URL whose path you want to test.
- Leave the rules field empty to fetch robots.txt, or paste a proposed file without publishing it.
- Read the matched rule for each crawler. Check firewall controls and actual server logs separately.
Method and limits
This is a robots.txt rule check, not a live test of a crawler’s network or indexing access. A rule that allows crawling does not prove a bot can pass a firewall, render a page, index it or cite it. Each named user agent is evaluated as a policy token, not impersonated. AI search and AI training crawlers can have different roles. Rules and product policies can change; consult the linked documentation.
Method maintained by Traffichood. Findings are rule-based observations of a public response, not a ranking or indexing prediction.
Questions about this check
Does Allow mean AI can access or cite my page?
No. Robots permission is one policy signal. Network access, firewall decisions, rendering, index selection and citations are separate matters. This tool does not measure them.
Is blocking a training crawler the same as blocking AI search?
No. Providers publish separate user-agent policies for different products and purposes. For example, OpenAI documents GPTBot and OAI-SearchBot separately. Check the provider’s current documentation before changing rules.
Will robots.txt keep sensitive pages private?
No. Robots rules are public instructions for cooperative crawlers. Protect private content with authentication and appropriate access controls.
Reference sources
- Google: robots.txt specification
User-agent groups, matching paths and rule precedence.
- RFC 9309: Robots Exclusion Protocol
The protocol used for robots.txt matching.
- OpenAI: crawler overview
Different roles of search, user-triggered and training user agents.
- Cloudflare: AI Crawl Control and WAF
Why firewall enforcement must be reviewed separately from robots rules.