robots.txt for AI bots: which to allow and which to block

TL;DR

• Sort AI bots by purpose, not by company: training crawlers, AI search crawlers and user-triggered fetchers. Each operator documents separate names for each.
• For most B2B sites, the sensible default is allow search, decide on training: allow OAI-SearchBot, Claude-SearchBot and PerplexityBot; block GPTBot, ClaudeBot and similar if you do not want to contribute training data.
• robots.txt is a request, not a lock. User-triggered fetchers often ignore it, and Google's AI Overviews cannot be blocked without leaving Google Search.

How robots.txt works, briefly

robots.txt is a plain-text file at the root of each host (https://www.example.com/robots.txt). The format is standardized in RFC 9309, which is blunt about its limits: "These rules are not a form of access authorization." Well-behaved crawlers read it and follow it; nothing forces them to.

Four rules from the standard and from Google's documentation cause most surprises with AI bots:

1. A crawler obeys the most specific group that names it. If your file has a User-agent: GPTBot group, GPTBot follows that group and ignores the User-agent: * group. Google's robots.txt documentation says it directly: "User agent specific groups and global groups (*) are not combined." Any Disallow you need for everyone must be repeated in each named group.
2. Within a group, the longest matching path wins. Allow: /docs/public/ beats Disallow: /docs/ for URLs under /docs/public/.
3. Changes take about a day. RFC 9309 says crawlers should not use a cached copy for more than 24 hours. OpenAI, Perplexity and Meta each say changes can take around 24 hours to apply.
4. A broken file can block everything. If robots.txt returns a server error (5xx), RFC 9309 says the crawler "MUST assume complete disallow." A missing file (4xx) means the crawler may fetch anything.

Each subdomain needs its own file. Anthropic, for example, tells site owners to repeat the rule on every subdomain they want excluded.

The AI crawlers that matter, by purpose

Training crawlers

These collect content that may be used to train models. Blocking them is the standard way to opt out of training, and it does not affect search visibility on the same operator's search products.

Note the catch with Google-Extended: because it also covers grounding — giving Gemini fresh Search results at answer time — blocking it can keep your pages out of answers in the Gemini app, not just out of training.

AI search crawlers

These build the indexes that AI answer products search and cite. Blocking them removes you from those answers.

Google has no separate AI Overviews crawler. Its AI features documentation says robots.txt rules for Googlebot are the control, and points to nosnippet, data-nosnippet, max-snippet and noindex for limiting what is shown. Microsoft's controls are also page-level: in a 2023 announcement, written when Copilot was called Bing Chat, Bing said content with a NOCACHE robots meta tag may appear in answers with only the URL, title and snippet, while NOARCHIVE content is not included in answers or used for training Microsoft's foundation models.

User-triggered fetchers

These fetch a page because a person asked a question or gave a task: ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, meta-externalfetcher. Most operators say robots.txt may not apply to them. Google says its user-triggered fetchers "generally ignore robots.txt rules" (Google). Anthropic is the exception: it says Claude-User honors robots.txt, and blocking it "prevents our system from retrieving your content in response to a user query." For a fuller picture of how these visitors show up in your logs, see how to tell when AI agents visit your website.

Three policies and the files that implement them

Policy A: be found, don't train (a common B2B default)

You want buyers' assistants to find and cite your pricing and docs, but you do not want to supply training data.

# Policy: appear in search and AI answers; opt out of model training.

# Training: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

# AI search: allow, minus private areas
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /app/
Disallow: /admin/

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /app/
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Google-Extended is left out on purpose, so Gemini can still ground answers in your pages. Add it to the first group if you prefer to opt out of Gemini training even at that cost. Note how /app/ and /admin/ are repeated: the search bots matched their own group, so they never read the * rules.

Policy B: open to all

You want maximum reach and are comfortable with training use. You still keep private paths out.

User-agent: *
Disallow: /app/
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

With no named groups, every crawler — AI or not — follows the same rules. This is also what you get if you never mention AI bots at all.

Policy C: keep content out of AI products

You publish premium content and want it out of AI training and AI answers.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /app/

Know the limits: listing user-triggered fetchers is a statement, and only some honor it. Google's AI Overviews still use Googlebot, so add nosnippet or max-snippet where needed, and use Bing's NOARCHIVE for Copilot. For hard enforcement, add firewall or bot-management rules.

Content Signals: saying how content may be used

robots.txt answers "may you fetch this?" but not "what may you do with it?". Cloudflare's Content Signals Policy, published in September 2025, adds a line for that, with three signals: search (building a search index and showing links and short excerpts), ai-input (using content as input to AI answers, such as retrieval and grounding) and ai-train (training or fine-tuning models).

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Cloudflare is clear that "content signals express preferences; they are not technical countermeasures against scraping," and that some companies may ignore them. In April 2026, Cloudflare found Content Signals on 4% of the top 200,000 domains. Adding one costs nothing and documents your intent.

Special cases on B2B sites

A few parts of a typical B2B site need their own thinking, whatever policy you pick.

• The app and the dashboard. Logged-in areas (/app/, /account/) should be disallowed for every crawler and, more importantly, protected by authentication. robots.txt keeps polite bots out; it does nothing for anyone else.
• Docs on a subdomain. docs.example.com reads its own robots.txt, not the one on www. Developer docs are often what buyers and coding assistants ask AI tools about, so it can make sense to allow AI search crawlers there even if you block training on the marketing site.
• Staging and preview sites. Do not rely on robots.txt to hide them; Google's documentation notes that a disallowed URL may still be indexed and shown without a snippet. Put previews behind a password or send a noindex header.
• Internal search and filter pages. URLs such as /search?q= or endless filter combinations waste every crawler's time. Disallowing them for all agents is usually right.
• Gated PDFs and whitepapers. If a document sits behind a form for people, do not leave the direct file URL open to crawlers. Either gate it properly or accept that it is public.
• Pricing and comparison pages. These are the pages buyers' assistants fetch most often. Make sure no rule written for a different purpose — a crawl budget rule, a bot challenge, a geo block — catches them by accident.

Common mistakes

• Forgetting that named groups ignore *. The most common bug in AI-specific robots.txt files. Repeat shared rules in every group.
• Blocking Googlebot to avoid AI Overviews. That removes you from Google Search entirely.
• Blocking Bingbot. That removes you from Bing and from Copilot answers that draw on Bing's index.
• Blocking every AI bot by reflex. Cloudflare's September 2026 network data shows fewer than 1% of sites on its network block search crawlers, while 17% block training. Most sites have decided that being cited is worth it.
• Relying on Crawl-delay. Google ignores it. Anthropic supports it as a non-standard extension. Use rate limiting at your server or CDN instead.
• Letting robots.txt fail. A 5xx on robots.txt can be read as "disallow everything." Monitor it like any other critical URL.
• Trusting the user agent alone. A request that says GPTBot may not be OpenAI. Verify against published IP ranges before you make exceptions.

How to roll out a change safely

robots.txt is one layer of being visible to AI on your own terms. The rest — content agents can read and forms they can use — is in the agent-ready website checklist. If you want to give agents a curated reading list as well, see llms.txt explained.

Revisit the file whenever an operator announces a new bot. The AI crawler list has changed several times since 2023, and a policy written per purpose ages much better than a list of names.