AI Crawler Checker
How to block GPTBot while allowing OAI-SearchBot
Express separate preferences for OpenAI search discovery and model-training crawling without replacing the rest of your robots.txt policy.
View solutionSolutions
Complete a specific web task.
Follow focused, source-backed steps for configuring crawler access, testing directives and diagnosing public website signals.
AI Search
AI Crawler Checker
How to check Claude-SearchBot access to your website
Test the crawler used for Anthropic search separately from ClaudeBot training collection and Claude-User requests.
View solutionCrawling & Indexing
HTTP Headers
How to check X-Robots-Tag and noindex directives
Inspect the final HTTP response for indexing and search-presentation directives that may not appear in the page HTML.
View solutionSitemap Explorer
How to inspect an XML sitemap index and its child sitemaps
Confirm that an index is valid, its child sitemap documents are reachable, and the listed URLs represent the canonical pages you intend search engines to discover.
View solutionRobots.txt Tester
Test robots.txt wildcard matching and rule precedence
Verify how a crawler and path match wildcard rules instead of inferring behavior from the file by eye.
View solutionWeb Diagnostics
CSP Debugger
How to debug Content Security Policy script-src errors
Trace a blocked script to the effective CSP directive, then test a narrow source, nonce or hash change instead of weakening the whole policy.
View solution