Test robots.txt wildcard matching and rule precedence

When wildcard testing matters

Use wildcard rules when a simple path prefix would be too broad: blocking one file extension, allowing a public subdirectory or matching a path ending. Test before deployment because a short pattern can affect many URLs, and crawler implementations do not all document identical edge behavior.

Understand the supported patterns

Google, Bing and other major search engines support a limited wildcard form. In Google’s interpretation, * matches any sequence of characters and $ designates the end of the URL.

User-agent: *
Disallow: /*.pdf$
Allow: /public/*.pdf$

For this example, the general rule blocks URLs ending in .pdf, while the longer public-path rule permits matching PDFs below /public/.

Test the example paths

Test path	Expected result	Reason
/manual.pdf	Disallowed	Matches /*.pdf$.
/public/guide.pdf	Allowed	The public Allow rule is more specific.
/manual.pdf?download=1	Not matched by the anchored PDF rule	The query string is part of the tested URL, so it does not end in .pdf.

The Google-compatible profile converts * to a variable sequence, honors a trailing $, compares rule specificity, and prefers Allow when equally specific matching rules conflict. It preserves path case, removes fragments and normalizes equivalent percent-encoded forms before comparison.

The following regression examples are also covered by Scopeva’s matcher tests:

Rule	URL	Result
Disallow: /*.php$	/file.php	Blocked
Disallow: /*.php$	/file.php?x=1	Not specified
Disallow: /*.php	/file.php?x=1	Blocked
Disallow: /*.php$	/file.php/	Not specified

Run an exact test

1. Open the Robots.txt Tester.
2. Fetch the production file or paste the candidate content.
3. Select the crawler whose behavior you need to evaluate.
4. Enter a path beginning with /, including a query string when it is part of the scenario.
5. Inspect the matched rule and status.
6. Repeat with a nearby path that should have the opposite result.

Testing both an intended match and a non-match catches broad patterns that one positive example can miss.

Specificity and conflicting rules

Google uses the most specific matching rule based on path length. If conflicting matching rules have equal specificity, Google uses the least restrictive rule. Scopeva follows the same practical Allow-first tie behavior for equal-length matches.

User-agent selection happens before path selection. Google-compatible mode selects the most specific matching product token, merges repeated groups with that same specificity and does not merge a specific group with User-agent: *. Test the actual crawler token rather than reading only the wildcard group.

Encoding, case and boundaries

Robots path matching is case-sensitive. Google-compatible mode compares canonical percent-encoded forms while keeping encoded reserved characters distinct. Query strings participate in matching; fragments do not. Test the exact URL form the crawler requests, and do not assume that /private means only one directory boundary unless the pattern expresses it.

Common mistakes

• Omitting $ when the intent is to match only a file ending.
• Treating * as a full regular-expression language.
• Comparing rule order instead of specificity.
• Ignoring the selected user-agent group.
• Testing a decoded or lowercased path that differs from the requested URL.
• Applying a Google-specific edge-case rule to every crawler without verification.

Limitations

Scopeva provides a deterministic interpretation for practical testing; it is not a substitute for every crawler’s private implementation. When a provider documents different syntax or behavior, follow that provider’s rules and verify with request logs where possible.