# AI crawlers and robots.txt: GPTBot, OAI-SearchBot, PerplexityBot

> In short: Separate search crawlers from training crawlers. Blocking OAI-SearchBot, PerplexityBot, Claude-SearchBot or Googlebot keeps you out of those assistants' answers; blocking GPTBot, ClaudeBot or Google-Extended only opts you out of training (and, for Google-Extended, Gemini's grounding) without affecting Google Search. Fetchers acting for a user, like ChatGPT-User and Perplexity-User may not follow robots.txt at all.

Guide from Keats, published 2026-11-13, updated 2026-11-13. Canonical: https://getkeats.com/guides/ai-crawlers-robots-txt

Every few months someone on a marketing team discovers that the site's robots.txt blocks "all AI bots", added in a hurry in 2023. Sometimes that was a deliberate choice about training data. Often it also, by accident, removed the site from AI search answers. The two are controlled by different user agents, and the difference matters.

## Three kinds of AI crawler

| Kind | What it does | If you block it |
|---|---|---|
| **Search crawler** | Reads pages so an assistant can find and cite them when it answers | You stop appearing in that assistant's answers |
| **Training crawler** | Collects pages that may be used to train future models | You opt out of training; answers that search the web are unaffected |
| **User-triggered fetcher** | Fetches a page because a person asked the assistant about it | Robots.txt may not apply |

## The user agents, by company

**OpenAI.** OpenAI documents three. **OAI-SearchBot** is for search: OpenAI says sites that are opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", though they can still appear as navigational links. **GPTBot** crawls content that may be used to train its foundation models. **ChatGPT-User** visits a page when a user's request needs it; OpenAI says that because these actions are initiated by a user, "robots.txt rules may not apply". OpenAI notes it "can take ~24 hours from a site's robots.txt update" for its search to adjust.

**Perplexity.** **PerplexityBot** surfaces and links sites in Perplexity's search results; Perplexity says it isn't used to train foundation models. **Perplexity-User** fetches pages when a user asks a question, and Perplexity says this fetcher "generally ignores robots.txt rules".

**Anthropic.** **ClaudeBot** collects training data. **Claude-SearchBot** "navigates the web to improve search result quality for users". **Claude-User** retrieves pages when a person's question to Claude needs them.

**Google.** There's no separate AI crawler for Google Search. AI Overviews and AI Mode are built on Google Search, so **Googlebot** governs them: a page has to be indexed and eligible for a snippet to appear. **Google-Extended** is a robots.txt token, not a separate crawler, that controls whether content is used for Gemini's training and grounding in some of Google's other systems. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". So blocking Google-Extended doesn't remove you from AI Overviews.

**Microsoft.** Bing's crawler, **Bingbot**, indexes for Bing. Microsoft's Bing Webmaster Tools now has an AI Performance report covering citations in Copilot and Bing's AI summaries, which is worth checking if Bing matters to your buyers.

## A robots.txt that allows answers but not training

If your aim is to appear in AI answers while opting out of training, this is the shape:

```
# Search: allow, so assistants can find and cite us
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

Leave Googlebot and Bingbot as they are for normal search. If you want to be cited everywhere, the simplest policy is to allow everything, which is what a robots.txt with no AI rules does.

Whether to opt out of training is a business decision, not an AEO one. Some publishers block it to protect content they license; most software and consumer brands want models to know their name and facts, and leave training open. What you should never do by accident is block the search crawlers.

## Check what's actually happening

- **Read your robots.txt** at yourdomain/robots.txt, including any rule for `User-agent: *` that could catch the bots above.
- **Check your CDN or firewall.** Bot protection at the network edge can block AI crawlers even when robots.txt allows them. Look for blocked requests by these user agents in your CDN's logs.
- **Check your server logs** for visits from each user agent. A search crawler that never shows up may be blocked further upstream.
- **Check rendering.** If your key facts only appear after JavaScript runs, some crawlers may not see them. Put prices and product facts in the HTML.

## What robots.txt can't do

It can't make an assistant name you, and it isn't a way to correct facts. Allowing the crawlers is the precondition; after that, what gets you named is the content and the third-party pages covered in [how ChatGPT search picks sources](https://getkeats.com/guides/chatgpt-search-sources). And an [llms.txt file](https://getkeats.com/glossary/llms-txt) is a different thing entirely: a map for agents, not a permission file. Our guide to [schema markup and llms.txt](https://getkeats.com/guides/schema-markup-ai-search) covers what each is good for.

For Google's controls over what appears in AI Overviews (nosnippet, data-nosnippet, max-snippet), see [AI Overviews vs AI Mode](https://getkeats.com/guides/ai-overviews-vs-ai-mode).


## Your agent does this

Ask your agent: "Check our robots.txt and tell me which AI assistants can read us for answers, and which only for training." It reads the file, compares each rule with the user agents on this page, and proposes the smallest change, for you to approve.

## Sources

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), read 2026-11-13
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots), read 2026-11-13
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read 2026-11-13
- [Google: Google's common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), read 2026-11-13
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), read 2026-11-13