---
title: AI crawlers explained for businesses
description: Which crawlers power search and AI answers, which only collect training data, and how to control each.
url: "https://poseidondssolutions.com/guides/ai-crawlers"
last_updated: 2026-11-18
publisher: Poseidon DS
---

1. [Home](https://poseidondssolutions.com/)
2. [Guides](https://poseidondssolutions.com/guides)
3. AI crawlers

Guide

# AI crawlers explained for businesses.

Some crawlers decide whether you can appear in search and AI answers. Others only collect training data. They are controlled separately.

By [Jeremy Williams](https://poseidondssolutions.com/about#founder), Founder & CEO · Facts last reviewed November 17, 2026 · [Editorial policy](https://poseidondssolutions.com/editorial-policy)

## The short version

- **Search crawlers** (Googlebot, Bingbot, Applebot) feed search engines — and Google’s AI Overviews and AI Mode are built on Google Search. ([source](https://developers.google.com/search/docs/appearance/ai-features))
- **AI search crawlers** (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed AI search answers and citations. Blocking them can keep you out of those answers. ([source](https://help.openai.com/en/articles/9237897-chatgpt-search-general-faq))
- **Assistant fetchers** (ChatGPT-User, Claude-User, Perplexity-User) open a page because a person asked about it.
- **Training crawlers and tokens** (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) only affect whether content may train AI models. Blocking them does not remove you from those companies’ search. ([source](https://developers.openai.com/api/docs/bots))

## Every crawler and token we track

| Name | Company | Used for | Purpose | How to verify |
| --- | --- | --- | --- | --- |
| Googlebot | Google | Search engine | Google Search, including AI Overviews and AI Mode | Reverse DNS name ends with googlebot.com or google.com · [docs](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) |
| Bingbot | Microsoft | Search engine | Bing Search and Microsoft Copilot answers | Reverse DNS name ends with search.msn.com · [docs](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) |
| Applebot | Apple | Search engine | Siri, Spotlight and Safari suggestions | Reverse DNS name ends with applebot.apple.com · [docs](https://support.apple.com/en-us/119829) |
| OAI-SearchBot | OpenAI | AI search | ChatGPT search results and citations | Published IP list: https://openai.com/searchbot.json · [docs](https://developers.openai.com/api/docs/bots) |
| ChatGPT-User | OpenAI | AI assistant fetch | Pages a ChatGPT user asks it to open | Published IP list: https://openai.com/chatgpt-user.json · [docs](https://developers.openai.com/api/docs/bots) |
| GPTBot | OpenAI | AI training | Content that may train OpenAI models | Published IP list: https://openai.com/gptbot.json · [docs](https://developers.openai.com/api/docs/bots) |
| Claude-SearchBot | Anthropic | AI search | Claude search result quality | Published IP list: https://claude.com/crawling/bots.json · [docs](https://support.claude.com/en/articles/8896518) |
| Claude-User | Anthropic | AI assistant fetch | Pages a Claude user asks it to read | Published IP list: https://claude.com/crawling/bots.json · [docs](https://support.claude.com/en/articles/8896518) |
| ClaudeBot | Anthropic | AI training | Content that may train Anthropic models | Published IP list: https://claude.com/crawling/bots.json · [docs](https://support.claude.com/en/articles/8896518) |
| PerplexityBot | Perplexity | AI search | Perplexity search results | Published IP list: https://www.perplexity.com/perplexitybot.json · [docs](https://docs.perplexity.ai/guides/bots) |
| Perplexity-User | Perplexity | AI assistant fetch | Pages a Perplexity user asks it to open (Perplexity says this fetcher generally does not follow robots.txt) | Published IP list: https://www.perplexity.com/perplexity-user.json · [docs](https://docs.perplexity.ai/guides/bots) |
| Google-Extended | — | AI training (robots.txt token only) | Gemini training and grounding uses (does not affect Google Search) | Not applicable — a rule, not a visitor |
| Applebot-Extended | — | AI training (robots.txt token only) | Apple foundation-model training | Not applicable — a rule, not a visitor |
| __usage_ai-training | — | AI training (robots.txt token only) | Shared AI-training preference (IETF draft, Oct 2026; providers are phasing in support) | Not applicable — a rule, not a visitor |

## An example robots.txt

This allows search and AI search, keeps admin pages private, and states a training choice. Replace the training lines with your own decision.

```
User-agent: *
Allow: /
Disallow: /admin

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://yourbusiness.com/sitemap.xml
```

robots.txt is a request that well-behaved crawlers follow, defined in RFC 9309. Names can be faked, so check the IP address or reverse DNS before trusting a visit. ([source](https://www.rfc-editor.org/rfc/rfc9309))

The full list is also available as data: [/data/ai-crawlers.json](https://poseidondssolutions.com/data/ai-crawlers.json).

## Want a second pair of eyes?

Run the free check on any business website, or ask for a full audit.

[Run the free check](https://poseidondssolutions.com/ai-readiness-check)[Request an audit](https://poseidondssolutions.com/contact)

## Sources

- [OpenAI — Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [OpenAI Help — ChatGPT search](https://help.openai.com/en/articles/9237897-chatgpt-search-general-faq)
- [Anthropic — Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518)
- [Perplexity — Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Apple — About Applebot](https://support.apple.com/en-us/119829)
- [Google Search Central — AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [RFC 9309 — Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)

Facts on this page were checked against the official sources below on Nov 17, 2026.

Questions

## Frequently asked questions

### Which crawlers decide whether I appear in AI answers?

AI search crawlers and fetchers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot for Google and Microsoft AI features.

### Does blocking GPTBot remove me from ChatGPT search?

OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate controls, so you can set each one on its own.

Last updated Nov 18, 2026

---

HTML version: https://poseidondssolutions.com/guides/ai-crawlers  
Site guide for AI assistants: https://poseidondssolutions.com/llms.txt  
Request an audit: https://poseidondssolutions.com/contact (people) or POST https://poseidondssolutions.com/api/v1/audit-requests (agents, see https://poseidondssolutions.com/openapi.json)
