Claude-SearchBot vs ClaudeBot vs Claude-User

Claude-SearchBot supports Anthropic’s search result quality, ClaudeBot collects public web content that may contribute to model training, and Claude-User retrieves content in response to a user’s request. Anthropic documents separate robots.txt controls for all three.

Key takeaways

If you are diagnosing one production URL, follow How to check Claude-SearchBot access.

• Use Claude-SearchBot for the search-discovery decision.
• Use ClaudeBot for the model-training decision.
• Use Claude-User for user-directed retrieval policy.
• Treat __usage_ai-training as a separate content-usage preference, not a crawler identity.
• Anthropic says its bots honor standard robots.txt directives and documents Crawl-delay support.
• Allowing access does not guarantee indexing, retrieval, citation, referral traffic or conversion.

Compare the Anthropic agents

Because the roles differ, a single company-wide assumption is not enough. A site can choose to support Claude search discovery while declining model-training collection, or make a separate choice for pages retrieved at a user’s direction.

Scopeva’s shared definitions are listed under Crawler types.

Example: allow search and user retrieval, block training

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

Anthropic says a restriction on Claude-SearchBot can reduce visibility and accuracy in user search results, while a restriction on ClaudeBot signals that future site material should be excluded from model-training datasets. Restricting Claude-User controls user-initiated retrieval separately.

Apply rules to the narrowest paths that meet the policy. Every subdomain needs its own root robots.txt file, and sensitive content still needs authentication.

AI Training Preference is not a crawler

Anthropic also recognizes the explicit __usage_ai-training policy token. It expresses a general AI-training usage preference under an Internet-Draft; it does not make HTTP requests and is not a universal crawler access block. The draft does not let User-agent: * substitute for this token.

Anthropic documents that an explicit ClaudeBot rule takes precedence over the general preference. Keep the two controls separate: ClaudeBot describes crawler access, while __usage_ai-training expresses intended content use.

Common mistakes

• Treating ClaudeBot as the token for every Claude-related request.
• Blocking ClaudeBot and assuming Claude-SearchBot or Claude-User is also blocked.
• Treating __usage_ai-training as a fourth Anthropic crawler.
• Applying a rule on the apex domain and assuming it governs every subdomain.
• Using robots.txt as a security boundary for private content.
• Adding Crawl-delay without monitoring whether the chosen rate solves the operational issue.
• Interpreting an Allowed result as a promise that Claude will retrieve or cite the page.

Check the policy with Scopeva

1. Run the AI Crawler Checker against the production page.
2. Review the three Anthropic rows independently.
3. Test representative paths with the Robots.txt Tester.
4. Verify the matched group using the robots evaluation rules.
5. Review diagnostic limitations before changing production access controls.

For a combined provider policy, use How to configure robots.txt for AI crawlers. For the equivalent OpenAI distinction, read OAI-SearchBot vs GPTBot vs ChatGPT-User.