Guide

AI crawlers explained for businesses.

Some crawlers decide whether you can appear in search and AI answers. Others only collect training data. They are controlled separately.

The short version

  • Search crawlers (Googlebot, Bingbot, Applebot) feed search engines — and Google’s AI Overviews and AI Mode are built on Google Search. (source)
  • AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed AI search answers and citations. Blocking them can keep you out of those answers. (source)
  • Assistant fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked about it.
  • Training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) only affect whether content may train AI models. Blocking them does not remove you from those companies’ search. (source)

Every crawler and token we track

NameCompanyUsed forPurposeHow to verify
GooglebotGoogleSearch engineGoogle Search, including AI Overviews and AI ModeReverse DNS name ends with googlebot.com or google.com · docs
BingbotMicrosoftSearch engineBing Search and Microsoft Copilot answersReverse DNS name ends with search.msn.com · docs
ApplebotAppleSearch engineSiri, Spotlight and Safari suggestionsReverse DNS name ends with applebot.apple.com · docs
OAI-SearchBotOpenAIAI searchChatGPT search results and citationsPublished IP list: https://openai.com/searchbot.json · docs
ChatGPT-UserOpenAIAI assistant fetchPages a ChatGPT user asks it to openPublished IP list: https://openai.com/chatgpt-user.json · docs
GPTBotOpenAIAI trainingContent that may train OpenAI modelsPublished IP list: https://openai.com/gptbot.json · docs
Claude-SearchBotAnthropicAI searchClaude search result qualityPublished IP list: https://claude.com/crawling/bots.json · docs
Claude-UserAnthropicAI assistant fetchPages a Claude user asks it to readPublished IP list: https://claude.com/crawling/bots.json · docs
ClaudeBotAnthropicAI trainingContent that may train Anthropic modelsPublished IP list: https://claude.com/crawling/bots.json · docs
PerplexityBotPerplexityAI searchPerplexity search resultsPublished IP list: https://www.perplexity.com/perplexitybot.json · docs
Perplexity-UserPerplexityAI assistant fetchPages a Perplexity user asks it to open (Perplexity says this fetcher generally does not follow robots.txt)Published IP list: https://www.perplexity.com/perplexity-user.json · docs
Google-Extended—AI training (robots.txt token only)Gemini training and grounding uses (does not affect Google Search)Not applicable — a rule, not a visitor
Applebot-Extended—AI training (robots.txt token only)Apple foundation-model trainingNot applicable — a rule, not a visitor
__usage_ai-training—AI training (robots.txt token only)Shared AI-training preference (IETF draft, Oct 2026; providers are phasing in support)Not applicable — a rule, not a visitor

An example robots.txt

This allows search and AI search, keeps admin pages private, and states a training choice. Replace the training lines with your own decision.

User-agent: *
Allow: /
Disallow: /admin

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://yourbusiness.com/sitemap.xml

robots.txt is a request that well-behaved crawlers follow, defined in RFC 9309. Names can be faked, so check the IP address or reverse DNS before trusting a visit. (source)

The full list is also available as data: /data/ai-crawlers.json.

Want a second pair of eyes?

Run the free check on any business website, or ask for a full audit.

Questions

Frequently asked questions

Which crawlers decide whether I appear in AI answers?

AI search crawlers and fetchers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot for Google and Microsoft AI features.

Does blocking GPTBot remove me from ChatGPT search?

OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate controls, so you can set each one on its own.

Last updated