Robots.txt Tester documentation

01

Fetch a site or paste a file

URL mode requests the public robots.txt file from the site's origin. Paste mode runs entirely in the browser and is useful for drafts or configurations that are not deployed yet.

A robots.txt file applies only to the protocol, host and port where it is served. Test the same origin that crawlers will visit.

02

How rule matching works

The tester selects the most specific matching user-agent group and then compares allow and disallow path rules. Longer matching paths take precedence, with allow winning when equally specific rules conflict.

• Test real URL paths rather than only the homepage
• Review wildcard and end-of-path patterns carefully
• Check every relevant crawler identity separately

03

What robots.txt cannot do

Robots.txt controls crawling requests; it is not an authentication or confidentiality mechanism. Public URLs may still be discovered through links, and different services may implement additional policies.

Use page-level robots directives when the goal is indexing control, and use access controls when content must remain private.