Short answer: Most AI companies use bots of three kinds: some collect content for training, some build the index behind AI search answers and some open a page because a person asked a question. Each has its own name in robots.txt, so you can allow or block them one by one.
Training, search and user bots
| Company | Training | Search | On a person’s request |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None, it says | PerplexityBot | Perplexity-User |
| Google-Extended token | Googlebot | None listed | |
| Apple | Applebot-Extended token | Applebot | None listed |
From each company’s page about its bots, read November 4, 2026. A token is a name for robots.txt, not a separate crawler. OpenAI also runs OAI-AdsBot, which only visits pages submitted as ads on ChatGPT.
OpenAI says its settings are independent: a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot, which signals that its content shouldn’t be used for training.
How robots.txt controls them
The crawlers read robots.txt. Anthropic says its bots honor it, and OpenAI and Perplexity say a change can take up to about a day to take effect.
Bots that act on a person’s request are different. OpenAI says robots.txt rules may not apply to ChatGPT-User, which isn’t used to decide what appears in search, and Perplexity says Perplexity-User generally ignores them.
Google-Extended and Applebot-Extended don’t crawl anything themselves. Google-Extended tells Google whether content it crawls may train Gemini or ground its answers; Applebot-Extended tells Apple whether it may train its models. Google says its token doesn’t affect inclusion in Search, and Apple asks for the nosnippet meta tag to keep content out of its AI answers.
Our robots.txt guide covers the file itself, and our llms.txt guide covers the other file some sites add for AI tools.
How to decide which AI bots to allow
- Decide what you want: appear in AI answers, be used for training, both or neither.
- Open your robots.txt and look for groups that name AI bots.
- To appear in ChatGPT search, allow OAI-SearchBot; to opt out of training, disallow GPTBot.
- Repeat for each company: ClaudeBot and Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended.
- If a firewall blocks bots, allow the IP ranges the companies publish for the bots you want.
- Wait about a day, then check that the change took effect.
- Look at your server logs to see which bots really visit, and check them against the published ranges.
How to tell a real AI bot from a fake one
Anyone can copy a bot’s name into a request. OpenAI, Anthropic, Perplexity, Apple and Google publish the IP addresses their bots use, so you can check whether a visit really came from them.
OpenAI and Perplexity recommend allowing those ranges if you want to appear in their search results. Anthropic warns that blocking its addresses may not work as an opt-out, because its bots then can’t read your robots.txt.
Which AI bots visited our site
We went through our server logs from October 24 to November 4, 2026, looking for these names. ClaudeBot, PerplexityBot, OAI-SearchBot and ChatGPT-User visited often, though less than Googlebot. GPTBot, the training crawler, came far less often.
Our robots.txt names no AI bot, so all of them may crawl the site.
Then we checked the addresses against the ranges OpenAI and Perplexity publish. Most OAI-SearchBot and PerplexityBot requests matched. Most requests that called themselves GPTBot did not, and none of those that called themselves Perplexity-User did.
So the names in a log are a starting point, not proof. We count a visit as a real AI bot only when its address matches the published list.
Before you change your robots.txt for AI
- You know which AI answers you want to appear in.
- Search bots you want are allowed, such as OAI-SearchBot.
- Training bots are allowed or blocked on purpose, not by accident.
- Google-Extended and Applebot-Extended are set the way you intend.
- Your firewall doesn’t block the published IP ranges.
- You check your logs against those ranges, not just the bot names.
RememberBlocking a training bot doesn’t hide you from AI search, and blocking a search bot does. Choose per bot.
AI search is part of our SEO work, along with local and technical SEO.
Quick answers about AI crawlers
What is an AI crawler?
A bot that an AI company sends to read web pages. Some collect content that may train models and some build the index behind AI search answers. Others fetch a page because a person asked the chatbot a question.
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI treats each bot separately: GPTBot is for training and OAI-SearchBot is for search. A site can disallow GPTBot and still appear in ChatGPT search answers, as long as OAI-SearchBot is allowed.
Does Google-Extended affect Google Search?
No. Google-Extended is a token in robots.txt, not a separate crawler. It controls whether content Google crawls may train Gemini or ground its answers, and Google says it doesn’t affect inclusion in Search.
What is ClaudeBot?
Anthropic’s crawler for content that may be used to train its models. Anthropic also runs Claude-SearchBot for search results and Claude-User, which fetches pages when someone asks Claude a question.
Can a bot ignore robots.txt?
Bots that act on a person’s request may. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Their crawlers follow robots.txt.
Sources
Read on November 4, 2026. OpenAI: Overview of OpenAI Crawlers. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? Perplexity: Perplexity Crawlers. Apple: About Applebot. Google: Google’s common crawlers. Our case: our own server logs from October 24 to November 4, 2026, and our robots.txt.