What are AI crawlers?

01

Crawler purposes differ

Search crawlers collect pages for indexes that answer later queries. Training crawlers collect material used to improve models. User-triggered agents may fetch a page in response to a live request. A provider can publish different user-agent names for these purposes.

Policies should therefore name the behavior you want to control rather than assuming one directive covers every product from the same company.

02

Signals a crawler can observe

Robots.txt is the first public policy signal most crawlers inspect. Page-level robots meta, X-Robots-Tag headers, canonical links, HTTP status and sitemap discovery provide additional context.

• Use explicit user-agent groups when policies differ
• Keep important public pages reachable through normal links
• Return stable status codes and avoid accidental redirect chains

03

Verify the exact deployment

Check the production protocol and hostname, not only a staging copy. CDNs can cache robots.txt, inject headers or redirect crawlers differently from local development.

An allowed result means the observed configuration permits a request. It does not promise indexing, citation, training use or inclusion in an AI answer.