How to block GPTBot while allowing OAI-SearchBot

When to use this solution

Use this configuration when you want public pages eligible for OpenAI’s search discovery while expressing that GPTBot should not crawl them for possible model-training use. OpenAI identifies OAI-SearchBot, GPTBot and ChatGPT-User as separate agents with different purposes.

The change is a crawl preference, not an access-control system. A server, CDN or WAF can still reject a request after robots.txt allows it, and robots.txt does not protect confidential content.

Before you start

Download or copy the current production robots.txt file before editing it. Inventory existing groups for Googlebot, Bingbot, other AI crawlers, private path exclusions and Sitemap declarations. The example below is a fragment to merge into that file, not a complete replacement.

Also decide whether your CDN or WAF has bot rules for OpenAI’s published IP ranges. A robots.txt rule and a network-layer allow or block rule describe separate layers; both can affect the observed result.

Configure the two crawler groups

Add explicit groups for the two automatic OpenAI crawlers:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

OAI-SearchBot is the search crawler. GPTBot crawls content that may be used to improve OpenAI’s generative AI foundation models. OpenAI states that these webmaster settings are independent, even though it may reuse the result of one fetch when both uses are allowed.

Do not add ChatGPT-User expecting it to control automatic search crawling. OpenAI says that agent performs user-triggered actions, may not be governed by robots.txt in the same way, and is not used to decide whether content can appear in Search.

Merge without overwriting existing policy

1. Keep every unrelated user-agent group and sitemap declaration already in production.
2. Add the two explicit groups as complete groups, separated from surrounding rules by a blank line.
3. Check whether an existing OAI-SearchBot or GPTBot group already exists. Edit that group instead of creating contradictory duplicates.
4. Test representative public and restricted paths, not only /.
5. Deploy to the hostname’s root at /robots.txt, then fetch the public file again to confirm the final response.

Do not assume a general User-agent: * group is merged into a specific crawler group. For example, Google’s parser selects the most specific matching user-agent group and does not combine it with the global group. Crawler implementations can differ at edge cases, so explicit rules are easier to review.

Verify the deployed result

Open the AI Crawler Checker with a public production URL. Inspect OAI-SearchBot and GPTBot as separate rows, then review the robots.txt source and matching rule shown by the result.

For path-level tests, paste or fetch the file in the Robots.txt Tester, select each crawler and enter a representative path. Confirm that OAI-SearchBot resolves to the intended Allow rule and GPTBot resolves to the intended Disallow rule.

An Allowed result only describes Scopeva’s interpretation of the observed robots.txt and fetch signals. OpenAI identities use Scopeva’s labeled generic compatibility profile because provider edge behavior may differ from Google’s parser. The result does not prove that OpenAI will index, retrieve, rank or cite the page.

Common mistakes

• Replacing the entire robots.txt file with the two-line example and losing existing policies.
• Blocking GPTBot and expecting that rule to remove a page from ChatGPT Search.
• Treating ChatGPT-User as the automatic Search crawler.
• Testing only the homepage while deeper paths have more specific rules.
• Allowing the crawler in robots.txt while a CDN, WAF or origin still blocks the request.
• Publishing sensitive URLs and relying on robots.txt to keep them private.

Limitations

Scopeva evaluates public responses at the time of the request. Cached robots.txt files, IP-based bot controls, geo rules, temporary upstream failures and provider-side processing can produce different behavior. Recheck after cache expiry and inspect server logs when you need evidence of real crawler requests.