Documentation

AI Crawler Checker documentation

The AI Crawler Checker requests the public signals a crawler can observe and explains how those signals apply to common search, training and user-triggered agents.

Last updated: 2026-10-16

What the checker inspects

Scopeva requests the submitted page, the site's robots.txt file, declared sitemap locations and selected response headers. It then evaluates each supported crawler against the most specific matching robots.txt group.

The check is based on public configuration. It does not sign in, execute privileged requests or attempt to bypass access controls.

Crawler-specific and wildcard user-agent groups
Allow and disallow rules for the submitted path
Sitemap declarations, redirects and X-Robots-Tag signals

How to read the result

Allowed means Scopeva did not find a matching rule that blocks the requested path. Blocked means a matching disallow rule won after applying user-agent and path specificity. Not specified means the crawler has no explicit policy in the observed robots.txt file.

These labels describe configuration, not guaranteed crawler behavior. A crawler may use additional policies, cached rules or product-specific controls that Scopeva cannot observe.

What to verify next

Review the matched rule before editing production configuration. If a result is unexpected, test the exact page path, check redirects and confirm that the robots.txt response is served from the site's origin root.

Keep important pages linked and present in a valid sitemap
Check page-level robots meta and X-Robots-Tag headers
Re-run the check after CDN or robots.txt caches have refreshed