How to check X-Robots-Tag and noindex directives

Know which control you are checking

Robots controls operate at different layers:

• robots.txt describes whether a crawler should fetch a path.
• A robots meta element is embedded in an HTML document.
• X-Robots-Tag is delivered in the HTTP response headers and can apply to HTML or non-HTML resources.

Do not diagnose a noindex problem by reading robots.txt alone. Inspect the final resource response as well as the page HTML where applicable.

Inspect the final response

A basic header looks like this:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Google also documents user-agent-specific forms:

X-Robots-Tag: googlebot: noindex

Follow the redirect chain and inspect the final response. A directive on an intermediate 301 response is not the same as a directive returned by the final 200 resource. CDNs and origins can each add headers, so check whether multiple X-Robots-Tag fields are present or combined.

Interpret common directives

noindex asks a supporting search engine not to show the resource in search results. nofollow asks it not to follow links from the resource. nosnippet restricts text or video snippets, while directives such as max-snippet set supported presentation limits.

Search engines do not necessarily support every directive identically. Use the documentation for the engine you are targeting and avoid treating one provider’s behavior as universal.

Check with Scopeva

1. Open HTTP Headers.
2. Enter the public URL, not only the origin hostname.
3. Review the redirect chain and final URL.
4. Confirm the final HTTP status.
5. Open the SEO header group and inspect every X-Robots-Tag value.
6. Compare the result with any robots meta element in the final HTML.

Scopeva shows the bounded redirect chain, then analyzes headers from the public final response; it does not treat an intermediate redirect’s headers as the final resource policy. It also does not emulate each search engine’s complete indexing system.

Check crawlability before relying on noindex

A crawler must fetch the resource to read a robots meta element or X-Robots-Tag. If robots.txt blocks the path, the crawler may not see the noindex response. For a URL already known to a search engine, that can leave the engine with only limited information about the blocked URL.

When the goal is deindexing, verify that the intended crawler can retrieve the response carrying noindex, then monitor the search engine’s own tools. Do not expose private content merely to deliver a noindex header; authentication is the correct protection for confidential resources.

Common mistakes

• Inspecting the origin response while a CDN changes the public header.
• Reading only the first redirect response.
• Looking for X-Robots-Tag in page source instead of response headers.
• Assuming robots.txt Disallow delivers a noindex instruction.
• Overlooking duplicate or user-agent-specific header values.
• Assuming every engine supports max-snippet or another presentation directive identically.

Limitations

Scopeva reports headers visible from its request location. Geo rules, cache variants, authentication and user-agent-dependent responses can differ. A correctly delivered directive still does not predict when a search engine will recrawl or update its index.