What is an AI crawler? Definition and user agents
On this page
An AI crawler is a bot run by an AI company that fetches web pages to train models, build an AI search index or answer a user's question in real time.
How it works
AI companies run separate bots for separate jobs, and each identifies itself with its own user agent token. The vendors' own documentation, as of October 2026:
| Company | Training | AI search index | Fetch for a user's request |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed (PerplexityBot is not used for model training) | PerplexityBot | Perplexity-User |
Sources: OpenAI, Anthropic, Perplexity.
The difference matters for robots.txtrobots.txt is a plain-text file at the root of a website that tells crawlers which paths they may fetch, following the Robots Exclusion Protocol (RFC 9309).. Training and indexing bots follow it. User-triggered fetchers are treated differently: OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a person initiated the request, and Perplexity says Perplexity-User "generally ignores robots.txt rules". Anthropic says its bots honor robots.txt.
Google works another way. Google-Extended is not a separate crawler but a robots.txt token that controls whether content Google crawls may be used to train Gemini models and to ground Gemini answers; Google says it does not affect inclusion or ranking in Google Search (Google).
Infrastructure providers have stepped in too. On July 1, 2025, Cloudflare announced it would block AI crawlers by default unless site owners allow them, and began asking every new domain at sign-up whether to allow AI crawlers (Cloudflare).
Why it matters for your website
Blocking "AI bots" as one group is rarely what a business wants. Blocking a training crawler keeps your content out of future model training. Blocking an AI search crawler can remove you from AI answers: OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Decide per purpose, and check your CDN or firewall settings too, since they may block bots before robots.txt is ever read.
Example
A B2B company that wants to appear in AI search answers but keep its content out of model training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Note that the Google-Extended rule also opts out of grounding in Gemini apps, which some companies may not want.
Related terms
- robots.txt: the file these rules live in.
- llms.txt: a proposed guide to your key pages for AI tools.
- Generative engine optimization (GEO): being found and cited by AI answers.
- Zero-click search: why AI answers can reduce visits.
- Agentic browser: agents that visit on behalf of one person.
Sources
- Overview of OpenAI crawlers (OpenAI API documentation) — accessed October 20, 2026
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (Claude Help Center) — accessed October 20, 2026
- Perplexity crawlers (Perplexity documentation) — accessed October 20, 2026
- Google's common crawlers: Google-Extended (Google Crawling Infrastructure) — accessed October 20, 2026
- Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large (Cloudflare, July 2025) — accessed October 20, 2026
Articles that use this term
Agent-ready web
AI agents visiting your site: how to tell and what they want
How to spot AI crawlers, user-triggered fetchers and browser agents in your logs, verify they are genuine, and what each kind of AI visitor wants from you.
Agent-ready web
How to make your website agent-ready: a practical checklist
A practical checklist for making a website readable and usable by AI agents: crawler access, content agents can parse, forms they can use and how to measure.
Agent-ready web
llms.txt, robots.txt and sitemaps: what each file tells AI
robots.txt sets permissions, sitemap.xml lists pages and dates, llms.txt offers a reading list. What each file tells AI systems and how to keep them in sync.