Guide
AI crawlers explained for businesses.
Some crawlers decide whether you can appear in search and AI answers. Others only collect training data. They are controlled separately.
The short version
- Search crawlers (Googlebot, Bingbot, Applebot) feed search engines — and Google’s AI Overviews and AI Mode are built on Google Search. (source)
- AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed AI search answers and citations. Blocking them can keep you out of those answers. (source)
- Assistant fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked about it.
- Training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) only affect whether content may train AI models. Blocking them does not remove you from those companies’ search. (source)
Every crawler and token we track
| Name | Company | Used for | Purpose | How to verify |
|---|---|---|---|---|
| Googlebot | Search engine | Google Search, including AI Overviews and AI Mode | Reverse DNS name ends with googlebot.com or google.com · docs | |
| Bingbot | Microsoft | Search engine | Bing Search and Microsoft Copilot answers | Reverse DNS name ends with search.msn.com · docs |
| Applebot | Apple | Search engine | Siri, Spotlight and Safari suggestions | Reverse DNS name ends with applebot.apple.com · docs |
| OAI-SearchBot | OpenAI | AI search | ChatGPT search results and citations | Published IP list: https://openai.com/searchbot.json · docs |
| ChatGPT-User | OpenAI | AI assistant fetch | Pages a ChatGPT user asks it to open | Published IP list: https://openai.com/chatgpt-user.json · docs |
| GPTBot | OpenAI | AI training | Content that may train OpenAI models | Published IP list: https://openai.com/gptbot.json · docs |
| Claude-SearchBot | Anthropic | AI search | Claude search result quality | Published IP list: https://claude.com/crawling/bots.json · docs |
| Claude-User | Anthropic | AI assistant fetch | Pages a Claude user asks it to read | Published IP list: https://claude.com/crawling/bots.json · docs |
| ClaudeBot | Anthropic | AI training | Content that may train Anthropic models | Published IP list: https://claude.com/crawling/bots.json · docs |
| PerplexityBot | Perplexity | AI search | Perplexity search results | Published IP list: https://www.perplexity.com/perplexitybot.json · docs |
| Perplexity-User | Perplexity | AI assistant fetch | Pages a Perplexity user asks it to open (Perplexity says this fetcher generally does not follow robots.txt) | Published IP list: https://www.perplexity.com/perplexity-user.json · docs |
| Google-Extended | — | AI training (robots.txt token only) | Gemini training and grounding uses (does not affect Google Search) | Not applicable — a rule, not a visitor |
| Applebot-Extended | — | AI training (robots.txt token only) | Apple foundation-model training | Not applicable — a rule, not a visitor |
| __usage_ai-training | — | AI training (robots.txt token only) | Shared AI-training preference (IETF draft, Oct 2026; providers are phasing in support) | Not applicable — a rule, not a visitor |
An example robots.txt
This allows search and AI search, keeps admin pages private, and states a training choice. Replace the training lines with your own decision.
User-agent: * Allow: / Disallow: /admin User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / Sitemap: https://yourbusiness.com/sitemap.xml
robots.txt is a request that well-behaved crawlers follow, defined in RFC 9309. Names can be faked, so check the IP address or reverse DNS before trusting a visit. (source)
The full list is also available as data: /data/ai-crawlers.json.
Want a second pair of eyes?
Run the free check on any business website, or ask for a full audit.
Sources
- OpenAI — Overview of OpenAI crawlers
- OpenAI Help — ChatGPT search
- Anthropic — Does Anthropic crawl data from the web?
- Perplexity — Perplexity crawlers
- Apple — About Applebot
- Google Search Central — AI features and your website
- RFC 9309 — Robots Exclusion Protocol
Facts on this page were checked against the official sources below on .
Questions
Frequently asked questions
Which crawlers decide whether I appear in AI answers?
AI search crawlers and fetchers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot for Google and Microsoft AI features.
Does blocking GPTBot remove me from ChatGPT search?
OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate controls, so you can set each one on its own.
Last updated