How to configure robots.txt for AI crawlers

01

Define the policy before the syntax

Separate search visibility, AI answer retrieval and model training in your policy discussion. Providers may publish distinct crawlers for those uses, and the names can evolve over time.

Document why each rule exists so future maintainers do not remove a deliberate restriction or keep an obsolete crawler group indefinitely.

02

Write narrow, testable rules

Start with the smallest path restriction that matches the requirement. Broad disallow rules can hide documentation, product pages or assets that are needed to understand the visible page.

• Place robots.txt at the origin root
• Use the published user-agent token exactly
• Test representative allowed and blocked paths
• Keep sitemap declarations absolute and current

03

Deploy and monitor

After deployment, fetch robots.txt through the same CDN and hostname crawlers use. Confirm the response is successful, plain text and not replaced by an application error page.

Re-test after redirects, CDN rules or domain migrations. A robots.txt rule cannot protect private information; sensitive content still requires authentication and access control.