Skip to content

Traffichood / Site diagnostics

← All SEO tools

Robots.txt and AI crawler checker

See which robots.txt rules apply to a page. Compare search and AI crawler policies, or test pasted rules before changing your live file.

Enter a public HTTPS page. Use a page you are authorized to check; do not include private tokens or personal data.

Leave empty to fetch the origin’s robots.txt, or paste rules to test a proposed change against the page URL. Maximum 65,536 characters.

Free basic access: 5 requests/day per network IP. Free Google accounts: 50/day per account. Both are limited to 6 requests/minute. Failed attempts and image loads count; daily limits reset at midnight UTC. Shared-network and site-wide safety limits may apply.

The service processes the URL or content you submit. URL checks request the public page. Avoid confidential URLs and content. Downloaded reports stay on your device; contact us only when you choose to share your findings. Privacy policy

Example to try

For a local rule example, use https://example.com/private/ and paste “User-agent: *” followed by “Disallow: /private/” on the next line. This tests a restriction without changing any website.

Your diagnostic report

Your report will appear here. Basic guest checks work without an account; Google sign-in unlocks the full free report.

  • 01 Enter the specific page URL whose path you want to test.
  • 02 Leave the rules field empty to fetch robots.txt, or paste a proposed file without publishing it.
  • 03 Read the matched rule for each crawler. Check firewall controls and actual server logs separately.

When to use it

For website owners investigating crawl restrictions and teams reviewing search, AI search or AI training crawler policies.

How to use this check

  1. Enter the specific page URL whose path you want to test.
  2. Leave the rules field empty to fetch robots.txt, or paste a proposed file without publishing it.
  3. Read the matched rule for each crawler. Check firewall controls and actual server logs separately.

Method and limits

This is a robots.txt rule check, not a live test of a crawler’s network or indexing access. A rule that allows crawling does not prove a bot can pass a firewall, render a page, index it or cite it. Each named user agent is evaluated as a policy token, not impersonated. AI search and AI training crawlers can have different roles. Rules and product policies can change; consult the linked documentation.

Method maintained by Traffichood. Findings are rule-based observations of a public response, not a ranking or indexing prediction.

Questions about this check

Does Allow mean AI can access or cite my page?

No. Robots permission is one policy signal. Network access, firewall decisions, rendering, index selection and citations are separate matters. This tool does not measure them.

Is blocking a training crawler the same as blocking AI search?

No. Providers publish separate user-agent policies for different products and purposes. For example, OpenAI documents GPTBot and OAI-SearchBot separately. Check the provider’s current documentation before changing rules.

Will robots.txt keep sensitive pages private?

No. Robots rules are public instructions for cooperative crawlers. Protect private content with authentication and appropriate access controls.

Reference sources