How to make your website agent-ready: a practical checklist

TL;DR

• "Agent-ready" means two things: AI crawlers and assistants can read your key facts, and agents acting for a person can use your site — find pages, fill forms, finish tasks.
• Most of the work is ordinary web hygiene: clear crawl rules, facts in server-rendered HTML, labeled forms, stable layouts. New pieces such as llms.txt and WebMCP are optional extras on top.
• When Cloudflare scanned the 200,000 most-visited domains in April 2026, few sites had done even the basics. That gap is an opportunity.

What "agent-ready" means

Your site now has two kinds of automated visitors that matter. The first is the AI crawler: a bot that fetches pages to build a search index or a training set, or to answer one user's question. The second is the agent that acts for a person — an assistant that opens your pricing page, compares plans and fills in the demo form, often inside an agentic browser. Google's AI optimization guide defines AI agents as "autonomous systems that can perform tasks on behalf of people" and notes that browser agents "may access your website to gather the data they need."

Cloudflare's September 2026 essay The Internet has a second audience puts the scale in numbers from its network: daily requests from AI agents grew by more than 1,700% over the past year, and the post says that this year, for the first time, more than half of Internet traffic wasn't human. The same post argues against blanket blocking: "Turn an agent away, and you may be turning away the person who sent it."

How ready is the web today?

Not very. In April 2026, Cloudflare Radar took the 200,000 most visited domains, filtered out categories where agent readiness matters little, such as redirect and ad-serving domains, and scanned the rest for its Agent Readiness score:

Most robots.txt files were written for classic search crawlers, not AI agents. The checklist below starts with what most sites can fix this week.

The checklist

1. Access: decide which bots may do what

• Write explicit robots.txt rules per AI crawler. Operators now split their bots by purpose. OpenAI, for example, documents GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for fetches a user asks for (OpenAI crawler docs). You can block training and still be citable in AI search. Our robots.txt guide for AI bots has a copy-ready file.
• Check your firewall, not just robots.txt. Bot-management rules, rate limits and CAPTCHAs can block crawlers you allowed in robots.txt. Anthropic, for example, states that its bots "will not attempt to bypass CAPTCHAs" (Claude Help Center).
• Review snippet controls. For Google's AI Overviews and AI Mode, a page "must be indexed and eligible to be shown in Google Search with a snippet" (AI features and your website). An old nosnippet or max-snippet:0 silently removes you.
• Consider Content Signals. A Content-Signal line in robots.txt states whether content may be used for search, AI answers or training. It expresses a preference; it does not enforce anything.

2. Content: put the facts in the HTML

• Server-render what matters. Pricing, plan limits, integrations, locations and contact details should be in the initial HTML, not injected by JavaScript after load or shown only in images. When Vercel analyzed AI crawler traffic in December 2024, it found that "none of the major AI crawlers currently render JavaScript," naming OpenAI's, Anthropic's, Meta's and Perplexity's bots; Google's Gemini, which uses Googlebot's infrastructure, was the exception (Vercel). Crawlers change, so recheck this, but server-rendered facts are safe either way.
• One clear page per question. If "Do you have a free plan?" is answered differently on three pages, assistants will pick one at random. Keep a single canonical page per fact and link to it.
• Date your facts. "Prices as of October 2026" on a pricing page helps any reader, human or model, judge freshness.
• Write for people. Google's guide says you don't need to write in a specific way for generative AI search, chunk pages into tiny pieces or add special schema. Clear, specific, original pages are the target.
• Keep structured data accurate if you use it. Google says it isn't required for generative AI features, but schema.org markup for your organization and products still helps rich results and must never contradict the visible page.
• Optional: offer Markdown. Some agents request Accept: text/markdown. Cloudflare can convert HTML to Markdown on the fly with Markdown for Agents, and reported that in February 2026 only 3 of 7 agents it tested sent that header by default.

3. Discovery: help agents find the right page

• Keep sitemap.xml complete and current, with lastmod dates you actually maintain.
• Link key pages from the main navigation and footer: pricing, product, docs, security, contact. Agents follow links like people do.
• Optional: publish an llms.txt that maps your most useful pages. Google Search ignores it, but some agents and tools read it; see llms.txt explained.

4. Interaction: make forms and buttons usable

• Label everything. Every input needs a real label, every button a text name, every icon button an accessible name. Lighthouse's experimental agentic browsing category includes "Accessibility for agents," which checks that "every interactive element has a programmatic name," that roles and parent-child relationships are valid, and that interactive content is not hidden from the accessibility tree (Lighthouse scoring).
• Keep the layout still. The same category tracks layout shift, because agents rely on where elements are. A banner that pushes the form down mid-click breaks agents as well as people.
• Avoid hostile patterns on low-risk paths. Modal pop-ups over pricing, CAPTCHAs on read-only pages and forms split across five steps are hard for agents and annoying for people.
• Try WebMCP on one form. Adding toolname and tooldescription to a form turns it into a WebMCP tool that agents can call (declarative API). It is a draft in origin trial, so it is cheap to try and safe to ignore if the API changes. Background: what is WebMCP.

5. Trust: keep humans in control of consequential actions

• Confirm before anything irreversible. Orders, cancellations, messages sent on a user's behalf: require an explicit human confirmation step.
• Validate on the server. An agent submitting a form is still untrusted input.
• Do not hand agents passwords. Cloudflare's readiness guidance prefers standard OAuth discovery over giving agents logged-in browser sessions, which it calls an "arguably unsafe workaround."
• Treat user-generated content with care. Reviews and comments on your pages can carry instructions aimed at agents. Keep them clearly separated from your own content.

6. Measurement: know who is visiting

• Read your logs for AI user agents. Declared crawlers name themselves (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) and most operators publish IP ranges to verify them.
• Use the AI Assistant channel in Google Analytics. Google Analytics now groups sessions referred by assistants such as ChatGPT, Gemini, Deepseek, Copilot or Grok into an "AI Assistant" channel; Google's own AI Overviews and AI Mode stay in Organic Search (GA4 channel definitions).
• Check Search Console. Google's guide points site owners to the Generative AI performance report in Search Console where it is available.

The facts agents look for on a B2B site

When an assistant researches a vendor for someone, the questions are predictable. Each one should have a single page that answers it in plain text, linked from your navigation:

Two habits make these pages work for machines as well as for people. First, state the answer in a sentence before the detail — "The Team plan costs [price] per user per month, billed annually" — instead of only showing it inside a pricing card graphic. Second, put the date of the last change on pages whose facts change, such as pricing and integrations. An assistant comparing your page with a two-year-old review needs a way to tell which is current.

If a fact is commercially sensitive and you do not publish it, say so: "Enterprise pricing is quoted on request." An explicit statement stops assistants from filling the gap with a guess from a third-party site.

A 30-minute audit you can run today

What not to do

• Don't block everything by reflex. Blocking search and user-triggered bots removes you from AI answers that buyers read. Block training if you want to; that is a separate switch.
• Don't show bots different content than people. Agents and search systems can compare the two, and a mismatch costs trust everywhere.
• Don't chase every new file format. Google's mythbusting section rejects AI-only files and rewrites for Google Search. Spend the time on the pages themselves first.

Where to start: effort versus payoff

None of this needs a rebuild. Work down the table, recheck every quarter, and keep the facts on your pages the single source of truth that every human and machine reader sees.