AI crawlers and robots.txt: GPTBot, OAI-SearchBot, PerplexityBot

Every few months someone on a marketing team discovers that the site's robots.txt blocks "all AI bots", added in a hurry in 2023. Sometimes that was a deliberate choice about training data. Often it also, by accident, removed the site from AI search answers. The two are controlled by different user agents, and the difference matters.

Three kinds of AI crawler

The user agents, by company

OpenAI. OpenAI documents three. OAI-SearchBot is for search: OpenAI says sites that are opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", though they can still appear as navigational links. GPTBot crawls content that may be used to train its foundation models. ChatGPT-User visits a page when a user's request needs it; OpenAI says that because these actions are initiated by a user, "robots.txt rules may not apply". OpenAI notes it "can take ~24 hours from a site's robots.txt update" for its search to adjust.

Perplexity. PerplexityBot surfaces and links sites in Perplexity's search results; Perplexity says it isn't used to train foundation models. Perplexity-User fetches pages when a user asks a question, and Perplexity says this fetcher "generally ignores robots.txt rules".

Anthropic. ClaudeBot collects training data. Claude-SearchBot "navigates the web to improve search result quality for users". Claude-User retrieves pages when a person's question to Claude needs them.

Google. There's no separate AI crawler for Google Search. AI Overviews and AI Mode are built on Google Search, so Googlebot governs them: a page has to be indexed and eligible for a snippet to appear. Google-Extended is a robots.txt token, not a separate crawler, that controls whether content is used for Gemini's training and grounding in some of Google's other systems. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". So blocking Google-Extended doesn't remove you from AI Overviews.

Microsoft. Bing's crawler, Bingbot, indexes for Bing. Microsoft's Bing Webmaster Tools now has an AI Performance report covering citations in Copilot and Bing's AI summaries, which is worth checking if Bing matters to your buyers.

A robots.txt that allows answers but not training

If your aim is to appear in AI answers while opting out of training, this is the shape:

# Search: allow, so assistants can find and cite us
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training: opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Leave Googlebot and Bingbot as they are for normal search. If you want to be cited everywhere, the simplest policy is to allow everything, which is what a robots.txt with no AI rules does.

Whether to opt out of training is a business decision, not an AEO one. Some publishers block it to protect content they license; most software and consumer brands want models to know their name and facts, and leave training open. What you should never do by accident is block the search crawlers.

Check what's actually happening

• Read your robots.txt at yourdomain/robots.txt, including any rule for User-agent: * that could catch the bots above.
• Check your CDN or firewall. Bot protection at the network edge can block AI crawlers even when robots.txt allows them. Look for blocked requests by these user agents in your CDN's logs.
• Check your server logs for visits from each user agent. A search crawler that never shows up may be blocked further upstream.
• Check rendering. If your key facts only appear after JavaScript runs, some crawlers may not see them. Put prices and product facts in the HTML.

What robots.txt can't do

It can't make an assistant name you, and it isn't a way to correct facts. Allowing the crawlers is the precondition; after that, what gets you named is the content and the third-party pages covered in how ChatGPT search picks sources. And an llms.txt file is a different thing entirely: a map for agents, not a permission file. Our guide to schema markup and llms.txt covers what each is good for.

For Google's controls over what appears in AI Overviews (nosnippet, data-nosnippet, max-snippet), see AI Overviews vs AI Mode.