How to check Claude-SearchBot access to your website

Understand the three Anthropic agents

Anthropic documents three crawler identities. Claude-SearchBot navigates the web to improve search result quality. ClaudeBot collects web content for model development. Claude-User supports user-initiated retrieval when a person asks Claude to access a website.

That distinction matters because a rule for one token is not automatically a rule for the others. Diagnose the agent connected to the behavior you want to control.

Before you start

Choose a real public URL and note its hostname and path. Robots rules are published per hostname, so a rule on www.example.com does not automatically control docs.example.com. Fetch the live /robots.txt instead of relying only on a local copy.

If you use Cloudflare, another CDN or a WAF, review bot and firewall policies separately. A crawler can be allowed by robots.txt and still receive a challenge, 403 response, connection failure or different content at the network layer.

Check the robots.txt group

A minimal explicit allow fragment is:

User-agent: Claude-SearchBot
Allow: /

If your policy is to decline model-training crawling separately, you can add:

User-agent: ClaudeBot
Disallow: /

These fragments are not a complete robots.txt file. Preserve existing groups, private path rules and sitemap declarations. Avoid duplicate crawler groups with conflicting rules.

Keep AI Training Preference separate

Anthropic also recognizes the __usage_ai-training policy token as a general preference about AI training use. It is not a crawler identity and does not itself block HTTP access. The mechanism is currently defined by an Internet-Draft rather than a finalized RFC.

User-agent: __usage_ai-training
Disallow: /

The draft treats this as an explicit content-usage preference; User-agent: * does not stand in for the policy token. Anthropic documents that an explicit ClaudeBot rule takes precedence over this general training preference, so keep crawler access policy and usage preference conceptually separate.

Test the target path

1. Fetch the public robots.txt file for the exact hostname.
2. Find the most specific group that applies to Claude-SearchBot.
3. Test the exact path, including relevant case and query-string behavior.
4. Record the matching Allow or Disallow rule rather than relying only on a status label.
5. Check the target URL’s final HTTP response after redirects.

Use the AI Crawler Checker to inspect crawler-specific results and the Robots.txt Tester to select Claude-SearchBot and test representative paths against pasted or fetched rules.

Separate robots policy from delivery access

Robots.txt describes crawler preference. It does not override a CDN or origin response. If Scopeva finds an Allow rule but the page fetch fails, inspect firewall events, managed bot rules, IP restrictions, rate limits, redirects and authentication.

Anthropic publishes crawler information in its official documentation. If you create network-layer allow rules, use the provider’s current data rather than copying an old third-party IP list.

Common mistakes

• Testing ClaudeBot when the question is about search discovery.
• Treating __usage_ai-training as a crawler or universal access-control directive.
• Assuming a general wildcard group overrides an explicit Claude-SearchBot group.
• Editing the robots.txt file on the wrong subdomain.
• Treating an HTTP 200 from robots.txt as proof that the target page is reachable.
• Describing an allowed result as guaranteed indexing or citation.
• Forgetting cached robots responses after deployment.

Limitations

Automated checks observe public responses from Scopeva’s network at one point in time. They cannot prove how Anthropic schedules crawls, processes content or selects sources. Geo-dependent WAF rules and provider-side cache behavior may differ from the check, so use origin/CDN logs for request-level confirmation.