AI agents visiting your site: how to tell and what they want

TL;DR

• AI visitors come in four kinds: training crawlers, search crawlers, user-triggered fetchers and browser agents. They want different things and follow different rules.
• Most declared bots name themselves in the user agent (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) and publish IP ranges you can check. Browser agents often look exactly like Chrome.
• The strongest signal, where it exists, is a cryptographic signature (Web Bot Auth). Combine logs, IP checks, signatures and analytics before you decide what to allow.

Four kinds of AI visitors

Not every AI crawler is doing the same job. Operators now document separate bots by purpose, and the split matters for both detection and policy.

The robots.txt column is where operators differ. Google says its user-triggered fetchers "generally ignore robots.txt rules" because a user requested the fetch (Google). OpenAI says that for ChatGPT-User, "robots.txt rules may not apply" (OpenAI). Perplexity says Perplexity-User "generally ignores robots.txt rules" (Perplexity), and Meta says its fetcher "may bypass robots.txt rules" (Meta). Anthropic, by contrast, says all three of its bots honor robots.txt (Claude Help Center).

Google is a special case on the crawler side. Google-Extended "doesn't have a separate HTTP request user agent string": Google crawls with its usual bots, and the token in robots.txt only controls whether content is used for Gemini training and grounding (Google). You will never see it in your logs.

How to tell: four levels of evidence

1. The user agent string

Declared bots put their name in the User-Agent header. In an access log they look like this (IP addresses replaced with documentation ranges):

203.0.113.10 - - [20/Oct/2026:09:14:02 +0000] "GET /pricing HTTP/1.1" 200 18234 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
203.0.113.27 - - [20/Oct/2026:09:15:40 +0000] "GET /docs/api HTTP/1.1" 200 40211 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
198.51.100.4 - - [20/Oct/2026:09:16:11 +0000] "GET /integrations HTTP/1.1" 200 22870 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"

A user agent is a claim, not proof. Anyone can send it. Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot" (Common Crawl), and the same applies to every well-known name.

2. Published IP ranges and reverse DNS

Most operators publish the addresses their bots use, so you can confirm a claimed identity:

A request that says GPTBot but comes from an address outside OpenAI's list is not OpenAI. Many CDNs and bot-management tools do this check for you and label such traffic as verified or not.

IP checks have limits. Cloudflare points out that IP-based identity is "brittle": addresses are shared across services and users, and they change when infrastructure changes (Cloudflare).

3. Cryptographic signatures (Web Bot Auth)

The newer approach is to have the agent sign each request. Web Bot Auth builds on RFC 9421, HTTP Message Signatures: the agent adds Signature and Signature-Input headers, and a Signature-Agent header that says where to fetch its public keys. The bot-specific parts are still IETF drafts, while RFC 9421 itself is a published standard.

Two examples of operators using it, as of October 2026:

• OpenAI says the cloud browser in ChatGPT Work "uses Web Bot Auth to sign outbound HTTP requests," with Signature-Agent set to "https://chatgpt.com" and public keys at chatgpt.com/.well-known/http-message-signatures-directory (OpenAI Help Center).
• Google says it is "experimenting with the Web Bot Auth protocol" for Google-Agent, using the identity https://agent.bot.goog.

A signed request looks roughly like this (values shortened):

Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1792487642;expires=1792487942;keyid="...";tag="web-bot-auth"
Signature: sig1=:Zm9vYmFy...:

You rarely verify these yourself. Cloudflare, Akamai and other edge providers check signatures and expose the result as a bot category. Cloudflare's documentation notes that as of July 1, 2026, it treats signed agents as verified and tracks whether a bot has a single operator ("Direct") or is "an agentic service that a wide range of end users can operate" ("Intermediary") (Cloudflare).

4. Behavior and referrals

Some agents leave no label at all. An agentic browser feature running in the visitor's own browser — Google's auto browse in Chrome, for example, which "handles multi-step chores on your behalf" (Google) — sends that browser's normal headers and cookies. To your server it is a regular visit. The clues, if any, are behavioral: very fast form completion, unusual navigation paths, no scrolling. They are not reliable enough to block on.

There is one place where you can get a clear signal from a browser agent: your own forms. If you mark a form up for WebMCP and an agent submits it through that interface, Chrome sets agentInvoked to true on the submit event. Logging that flag next to your normal form analytics tells you how many submissions came through an agent, without guessing from behavior. It only covers agents that use WebMCP, which as of October 2026 is an origin trial, so treat it as a sample, not a census.

The human side of AI traffic is easier to see. Google Analytics assigns sessions referred by assistants such as ChatGPT, Gemini, Deepseek, Copilot or Grok to an "AI Assistant" channel, while Google's own AI Overviews and AI Mode count as Organic Search (GA4 channel definitions). When an app strips the referrer, the visit lands in Direct.

What each kind of visitor wants

• Training crawlers want lots of text and will fetch widely. They give nothing back directly. Whether to allow them is a business decision about your content, not a technical one.
• Search crawlers want the pages an AI search product can cite: product, pricing, docs, comparisons. Blocking them removes you from answers. OpenAI, for example, says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."
• User-triggered fetchers want one page, now, to answer one person's question. They need fast responses and content in the HTML. A 403 or a bot challenge here means a buyer's question went unanswered.
• Browser agents want to finish a task: compare plans, fill a form, book a call. They need labeled fields, stable layouts and no pop-ups in the way. This is where WebMCP fits: it lets your page hand the agent named actions instead of making it guess.

The volume is real. Cloudflare's September 2026 network data shows daily AI agent requests up more than 1,700% in a year, and requests with a stated training purpose rising from 22% of crawler requests in spring 2025 to 52% by June 2026. Site owners already treat the two very differently: fewer than 1% of sites on Cloudflare block search crawlers, while 17% block training.

A starting policy by visitor type

Once you know who is visiting, set a policy per purpose rather than per company. A reasonable default for a B2B site that wants to be found and cited:

Write the crawler parts of this policy into robots.txt and the rest into your firewall or CDN rules, then recheck the logs a month later to confirm each group gets the response you intended.

A log review you can do this week

grep -oEi "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|google-agent|meta-externalagent|meta-externalfetcher|ccbot" access.log \
  | tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn

Common misreads

• "Googlebot is classic search, so it isn't AI." Google's own AI features documentation says AI is built into Search and that robots.txt rules for Googlebot are the control for AI Overviews and AI Mode. The same crawl feeds both.
• "No GPTBot hits means OpenAI isn't reading us." OpenAI notes that if a site allows both of its crawlers, "we may use the results from just one crawl for both use cases." Look for OAI-SearchBot and ChatGPT-User too.
• "A visit from chatgpt.com is a bot." A referral in analytics is a person who clicked a link in an answer. The bot fetch that produced the answer is in your server logs, not in analytics.
• "Blocking in robots.txt stops all AI access." It stops crawlers that honor it. User-triggered fetchers from several operators may not apply it, and browser agents are ordinary visits.

Where sources disagree

The line between "bot" and "agent acting for a person" is contested. In August 2025, Cloudflare reported that when Perplexity's declared bots were blocked, it observed undeclared crawlers fetching the same sites with a generic Chrome user agent, and removed Perplexity from its verified bot list (Cloudflare). Perplexity disputed the analysis and argued that user-driven agents, which fetch a page only when a person asks, should not be treated like crawlers (Perplexity).

You do not need to settle that debate to run a website. Decide what you allow per purpose, verify identities where you can, and keep the pages buyers' agents read accurate. The policy side is covered in robots.txt for AI bots; the readiness side in the agent-ready website checklist.

Every AI agent that reaches your site was sent by someone: a company building a model, a search product, or a person with a question. Knowing which is which is the first step to serving the ones that bring you business.