llms.txt explained: who reads it and how to write one

TL;DR

• llms.txt is a plain Markdown file at /llms.txt that gives AI agents a short, curated map of your site: who you are, what you offer and which pages to read for details.
• It is a community proposal, not a standard. Version 2 was published in August 2026.
• Coding agents and documentation tools use it, and Chrome's Lighthouse checks for it. Google Search says it ignores the file. Write one if it takes you an hour, but do not expect rankings from it.

What llms.txt is

llms.txt was proposed by Jeremy Howard on September 3, 2024, and is maintained at llmstxt.org. The idea is simple. Web pages are built for people, with navigation, scripts and banners around the actual information. An agent working for a user has a limited context window and pays for every wasted token. So the site owner publishes one small Markdown file that says, in effect: "here is what this site is, and here are the pages worth reading."

The format is deliberately strict so programs can parse it as well as models can read it. In order, a file contains:

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote with a short summary of the essentials.
3. Optional paragraphs or lists with more context (no headings).
4. Optional H2 sections with "file lists": Markdown links, each with an optional note after a colon.

By convention, a section titled ## Optional holds secondary links that an agent can skip when it is short on context.

What changed in version 2

The v2 change log (August 2026) is mostly about discoverability and matching real-world use:

• Markdown versions of pages. Pages can offer a clean Markdown copy at the same URL with .md appended (page.html.md) or replacing the extension (page.md). Both forms are now allowed.
• Link relations. A page can point to its Markdown version with rel="alternate" type="text/markdown" and to the llms.txt that covers it with rel="describedby", as HTML link elements or an HTTP Link: header.
• Subpath files. /docs/llms.txt covers everything under /docs/, and where several files apply, the most specific one wins.
• "Optional" is just a convention now. v1 tooling used it mechanically to decide what to drop. v2 says agents should read or search the file and follow links, so the section no longer carries special meaning.

Who actually reads llms.txt (as of October 2026)

This is where most articles get vague, so here is the evidence by reader, with sources.

Coding agents and documentation platforms: yes

The proposal says llms.txt files "are used most heavily for software documentation, where coding agents follow them to find API references and tutorials." The AI labs publish them for their own developer docs — see Anthropic's and OpenAI's. Several docs platforms and CMS plugins generate one automatically. If you sell software with an API or docs, this is the strongest reason to have one.

Google Search: no

Google's guide to generative AI features in Search is explicit: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities)." It adds that Google Search itself doesn't use them, and that creating them "will neither harm nor help your site's visibility or rankings in Google Search." AI Overviews and AI Mode draw on the same Search index as classic results.

Chrome Lighthouse: it checks

Lighthouse's experimental agentic browsing category includes an llms.txt audit. It describes the file as "a machine-readable summary of a website's content, specifically designed for LLMs and AI agents" and argues that without it, "agents may spend more time crawling the site." A missing file (404) is marked not applicable, because providing one "is optional at the moment"; a server error is flagged. The category's scoring page labels the whole category experimental.

AI crawlers: weak evidence

In April 2025, Google's John Mueller said that, as far as he knew, none of the AI services had said they use llms.txt, and that "you can tell when you look at your server logs that they don't even check for it." He compared it to the old keywords meta tag (Search Engine Journal). Cloudflare left llms.txt out of the default scan of its Agent Readiness score (you can add it as a custom check), while still recommending well-structured llms.txt files for large documentation sites.

Where the sources disagree

Google's two teams appear to contradict each other, and Search Engine Journal noted the split in May 2026. Both positions can be true at once: Search ranking does not use the file, while a browser agent that is already on your site might.

Should you write one?

Use this rule of thumb:

• Yes, prioritize it if you have developer docs, an API, integrations or a product where details (limits, plans, compatibility) matter and change often.
• Yes, but keep it small if you are a B2B site with a handful of key pages. It costs an hour and gives agents a clean summary of your offer.
• Don't expect search rankings, AI Overview placement or citations in chat answers from the file alone. Those come from the pages themselves being crawlable, indexed and useful. Our guide to robots.txt for AI bots covers the crawl side.

How to write an llms.txt file

A template for a B2B SaaS site

Here is a complete example for a fictional company. Copy the structure, not the claims.

# Example Scheduling

> Example Scheduling is self-serve meeting-booking software for B2B sales teams.
> Plans are priced per seat, billed monthly or annually, and a free plan covers one user.

Key facts:

- Works with Google Calendar and Microsoft 365 calendars.
- Data is hosted in the EU and the US; customers choose the region at signup.
- We do not offer on-premise installation.

## Product

- [Product overview](https://www.example.com/product.md): What the app does and who it is for
- [Routing rules](https://www.example.com/product/routing.md): How meetings are assigned to reps

## Pricing

- [Pricing](https://www.example.com/pricing.md): Plans, seat prices, limits and what each plan includes

## Docs

- [Getting started](https://www.example.com/docs/start.md): Install, connect a calendar, publish a booking page
- [API reference](https://www.example.com/docs/api.md): REST endpoints, authentication and rate limits

## Company

- [Security](https://www.example.com/security.md): Certifications, data retention and subprocessors
- [Contact](https://www.example.com/contact.md): How to reach support and billing

## Optional

- [Changelog](https://www.example.com/changelog.md): Release notes by month

And the matching Link header from the v2 proposal, which lets an agent on any page find both the Markdown copy and the llms.txt that covers it:

Link: </pricing.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="describedby"

Check that the file is actually reachable

A file that exists on disk but returns a login page, a bot challenge or the wrong content type is invisible to agents. Check it from the command line:

curl -sI https://www.example.com/llms.txt

You want three things in the response:

• Status 200, with no redirect to a login or consent page.
• Content-Type: text/plain or text/markdown, not text/html. Some frameworks route unknown paths to an HTML error page that still returns 200.
• No bot challenge. If you use a firewall or bot-management product, confirm that its rules let declared AI agents fetch the file. A file that only humans can load defeats the purpose.

Lighthouse flags server errors on the file, so running the agentic browsing category on your homepage is a quick second check.

What about llms-full.txt?

Many documentation sites also publish llms-full.txt: the full text of every docs page in one large file. It is a convention that grew up around the proposal, not part of the llms.txt format itself. Cloudflare, for example, recommends making llms.txt and llms-full.txt easy to find in documentation navigation. It makes sense when someone may want to load a whole manual into a coding tool. For a marketing site with a dozen key pages, the regular file with good links is enough.

Common mistakes

• Pasting the sitemap. A list of 2,000 URLs is not a map. The proposal wants a curated file that stays small enough to fit in context, with detail behind links.
• Marketing copy instead of facts. "Best-in-class platform" tells an agent nothing. "Free plan: one site, 100 pages" does.
• Saying something your pages don't. Mueller's keywords-meta-tag comparison was exactly this risk: a file where the owner claims things about the site. Any agent can check the pages, so the file must match them.
• Broken or blocked links. Links behind a login, blocked in robots.txt or returning 404 waste the agent's time. Check them every time you publish.
• Set and forget. Pricing and plan pages change. Regenerate the file from the same source as your pages if you can, or add it to your release checklist.

llms.txt, robots.txt and sitemap.xml side by side

Here is how the three root files compare.

They complement each other. A page blocked in robots.txt should not be in llms.txt, and llms.txt is no substitute for a sitemap. For the full picture of what makes a site easy for agents to read and use — including forms agents can call through WebMCP — see the agent-ready website checklist.

An llms.txt file is cheap to write and easy to keep honest if it comes from the same source as your pages. Treat it as a well-organized index for the agents that read it, not as a ranking trick, and you will not be disappointed by it.