A lot of sites blocked “AI” in 2023 with one line in robots.txt and never looked at it again. Two years on, the same line quietly removes them from ChatGPT search, Perplexity and Claude’s search results, because the crawlers that power answers are different tokens from the ones that gather training data. It is the cheapest AI SEO mistake to make and the cheapest to fix. This article lists every token, quotes what its owner says it does, and ends with a robots.txt you can copy.
OpenAI
OpenAI documents its agents on its bots page. Three of them matter for a business website.
| Token | What OpenAI says it does | robots.txt |
|---|---|---|
GPTBot | Used to “crawl content that may be used in training our generative AI foundation models”. | Disallowing it “indicates a site’s content should not be used in training”. It does not affect search answers. |
OAI-SearchBot | Used to “surface websites in search results in ChatGPT’s search features”. | Sites that opt out “will not be shown in ChatGPT search answers”. |
ChatGPT-User | Used for “certain user actions in ChatGPT and Custom GPTs. When users ask ChatGPT or a CustomGPT a question, it may visit a web page.” | “Because these actions are initiated by a user, robots.txt rules may not apply.” |
The common mistake is a single Disallow: / for GPTBot written in 2023, which is fine, sitting next to nothing at all for OAI-SearchBot, which is also fine, because unlisted agents fall through to the * rules. The problem is sites that added a blanket block for every OpenAI token when only one of them was ever about training.
Anthropic
Anthropic’s help article lists three agents and states that all three respect robots.txt.
| Token | What Anthropic says it does |
|---|---|
ClaudeBot | “Helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” |
Claude-SearchBot | “Navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.” |
Claude-User | “Supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.” |
Perplexity
Perplexity’s bots guide is explicit that its crawler is not a training crawler.
| Token | What Perplexity says it does | robots.txt |
|---|---|---|
PerplexityBot | “Designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” | Disallowing it removes the site from Perplexity search results. |
Perplexity-User | “Supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.” | “Generally ignores robots.txt rules” because the request is a user action. |
Google is the one where people over-correct. AI Overviews and AI Mode are Search features, and Google’s AI features documentation says eligibility is the ordinary requirement of being indexed and snippet-eligible, controlled by the usual nosnippet, max-snippet and noindex mechanisms. The separate Google-Extended token is described there as the control “to limit AI training and grounding in some of Google’s other systems”. Blocking Google-Extended does not remove you from Search or from AI Overviews; blocking Googlebot does.
A robots.txt that keeps you citable
This is the shape SearchLift recommends for a commercial site that wants to be found and cited but has a policy against training use. Adjust the training lines to your own position; the search and user-action lines are the ones that decide whether you appear in answers.
# Search and answer crawlers: allow, so you can be cited
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# User-action fetchers: allow; several ignore robots.txt anyway
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /
# Training crawlers: your policy decision. These lines block training only.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlTwo cautions. First, robots.txt rules match by longest path, and a group only applies to the exact user-agent it names, so a blanket Disallow under * still lets the named groups through; Google’s robots.txt reference describes the precedence rules. Second, the file only governs crawlers that read it; it is not a security control and it does nothing for content that is already in a model.
How to check your own file
Open /robots.txt on your domain and search it for each token above. If a search or user-action token is disallowed, or if a * block disallows everything and none of the search tokens are named, you are not citable by that assistant regardless of how good your pages are. For ChatGPT SEO specifically, OAI-SearchBot is the token that decides it. SearchLift’s audit checks this on every run and, where the platform lets it edit robots.txt, opens the tokens for you; elsewhere it gives you the exact lines.
Common questions
- 01Does blocking GPTBot remove me from ChatGPT search?
- No. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the search crawler; opting out of OAI-SearchBot is what removes a site from ChatGPT search answers.
- 02Which AI agents ignore robots.txt?
- By their owners’ own documentation, user-action fetchers may not honour it: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User “generally ignores” them, because a person triggered the fetch. Anthropic states all three of its agents respect robots.txt.
- 03Does Google-Extended affect AI Overviews?
- Google describes Google-Extended as a control for AI training and grounding in some of its other systems. Eligibility for AI Overviews and AI Mode follows ordinary Search indexing and snippet rules, which are governed by Googlebot and the snippet controls.
- 04Is robots.txt enough to stop AI training on my content?
- It asks crawlers that read it to stay out, and the documented training crawlers say they honour it. It is not a technical barrier and has no effect on content already collected.