llms.txt, robots.txt and sitemaps: what each file tells AI

TL;DR

• robots.txt says who may fetch what. It is a request that well-behaved crawlers honor, not a lock.
• sitemap.xml says which URLs exist and when each one last changed. Search engines, including the ones behind AI answers, use it to find and refresh pages.
• llms.txt says what your site is and which pages to read first. It is a proposal, read by some agents and ignored by Google Search.
• The three files only help when they agree with each other. A page blocked in one and promoted in another sends mixed signals to every machine that reads them.

Three files, three different questions

Every website has two audiences now: people, and software that reads pages on their behalf. Before an AI system can quote your pricing page, three things must happen. It has to learn the page exists, it has to be allowed to fetch it, and it has to understand what the page is about. Each of the three root files answers one of those questions.

They are not interchangeable. A sitemap grants no permission, robots.txt describes no content, and llms.txt neither blocks anything nor gets a page indexed. The rest of this guide covers what each file tells AI systems, where sites usually get them wrong, and how to check that they agree.

How an AI answer reaches your page

It helps to picture the path a page takes before it shows up in an answer from ChatGPT, Gemini, Copilot or Perplexity:

1. Discovery. A crawler learns that a URL exists, from links, from your sitemap or from a push notification such as IndexNow.
2. Permission. Before fetching, the crawler checks robots.txt for rules that apply to its user agent.
3. Fetching and parsing. The crawler downloads the HTML. Many AI crawlers do not run JavaScript, so content that only appears after scripts run may be invisible to them.
4. Indexing. The page goes into a search index or another store that AI systems retrieve from.
5. Answering. When someone asks a question, the system retrieves matching pages and writes an answer from them. Some assistants also fetch a page live because a user pasted a link or asked about it.

robots.txt sits at step 2, the sitemap at step 1, and llms.txt mostly at step 5, when an agent is already working on a task and wants a shortcut to the right pages. Step 3 is where content in the HTML matters: Vercel's 2024 study of AI crawler traffic found that "none of the major AI crawlers currently render JavaScript," with Google's Gemini and Applebot as the exceptions because they render pages like a browser.

robots.txt: permission, per user agent

The robots.txt format is defined by RFC 9309. It groups rules by user agent, and the most specific matching rule wins. The RFC is blunt about its limits: "These rules are not a form of access authorization." A crawler that ignores the file can still fetch your pages, so anything private belongs behind a login.

What it tells AI systems

For AI, robots.txt does two jobs.

It separates uses. Most AI companies now run several AI crawlers with different purposes, so one company can be allowed for search and blocked for training. OpenAI documents that "Each setting is independent of the others": OAI-SearchBot decides whether a site can appear in ChatGPT search answers, and GPTBot covers content "that may be used in training our generative AI foundation models." Google offers a control token rather than a crawler: Google-Extended governs use in Gemini model training and grounding in Gemini Apps, and Google states it "does not impact a site's inclusion in Google Search."

It can carry usage preferences. Cloudflare's Content Signals Policy adds a line such as Content-Signal: search=yes, ai-train=no to robots.txt. Cloudflare is clear that these "express preferences; they are not technical countermeasures against scraping."

What it does not control

User-triggered fetches are a gray zone. When someone asks ChatGPT about a page, it may visit with the ChatGPT-User agent, and OpenAI says that "because these actions are initiated by a user, robots.txt rules may not apply." Blocking a training crawler also does not remove you from AI search, and allowing a search crawler does not guarantee you will be cited.

Two technical details catch site owners out. Crawlers may cache your robots.txt for up to 24 hours, so changes are not instant. And if /robots.txt returns a server error (a 5xx status), the RFC says a crawler "MUST assume complete disallow." A broken robots.txt can quietly remove you from AI search for as long as it stays broken.

For a full list of AI user agents and three ready-made policies, see our guide to robots.txt for AI bots.

sitemap.xml: the inventory and its timestamps

The sitemap protocol lists the URLs you want search engines to know about. Each file holds at most 50,000 URLs and 50 MB uncompressed, and larger sites use a sitemap index that points to several files. You can announce it with one line anywhere in robots.txt (Sitemap: https://www.example.com/sitemap.xml) and submit it in Google Search Console and Bing Webmaster Tools.

What it tells AI systems

A sitemap tells crawlers two things: these are the pages that matter, and this is when each one last changed. The second part matters more than it used to.

• Google "ignores <priority> and <changefreq> values" and uses <lastmod> only "if it's consistently and verifiably accurate," according to its sitemap guide. It also notes that a sitemap is "merely a hint."
• Bing wrote in July 2025 that "the lastmod field in your sitemap remains a key signal, helping Bing prioritize URLs for recrawling and reindexing," and tied this directly to AI answers in Copilot (Bing Webmaster Blog). Bing also ignores changefreq and priority, and asks for ISO 8601 dates that include the time.

Freshness is the practical point for B2B sites. If you change a price, a plan limit or a list of integrations, an AI answer built from an old copy of the page will repeat the old fact. An accurate lastmod is how you ask search engines to fetch the new version sooner.

Pushing changes with IndexNow

A sitemap waits to be read. IndexNow lets you notify participating search engines the moment a URL is "added, updated, or deleted," and the IndexNow documentation says submitted URLs "will be automatically shared with all other participating search engines." Bing recommends using both: sitemaps for complete coverage, IndexNow for speed. Google does not list IndexNow among its sitemap submission methods, so keep the sitemap even if you adopt it.

llms.txt: a reading list for agents

llms.txt is a Markdown file at the root of your site with an H1, a short summary and lists of links to your most useful pages. Version 2 of the proposal was published in August 2026. The proposal positions it alongside the other two files: "While sitemaps list all pages for search engines, llms.txt offers a curated overview for LLMs," and it "can complement robots.txt by providing context for allowed content."

What it tells AI systems

llms.txt tells an agent what your company is and which pages answer which questions, in a format that costs few tokens to read. The proposal says the file is "used on demand, when an agent needs information about a topic while assisting a user." In other words, it serves step 5 above, not crawling.

What it does not do

It grants and blocks nothing, and it does not get pages indexed. Google's guide to generative AI features in Search says you don't need "new machine readable files, AI text files, markup, or Markdown," and that Google Search ignores them. Treat llms.txt as a courtesy to agents that read it, not as a ranking signal. Our llms.txt guide covers the format and who reads it in detail.

A matching set for one site

Here are the three files for a fictional B2B software company, written to agree with each other. The policy: allow search and user-triggered fetches, opt out of model training, keep the app and internal search results out of all crawlers.

# https://www.example.com/robots.txt
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /app/
Disallow: /search

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/pricing</loc>
    <lastmod>2026-10-14T09:30:00+00:00</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/integrations</loc>
    <lastmod>2026-09-02T16:05:00+00:00</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/security</loc>
    <lastmod>2026-06-20T11:00:00+00:00</lastmod>
  </url>
</urlset>

# Example Analytics

> Example Analytics is self-serve product analytics for B2B SaaS teams.
> Plans are priced per tracked user; a free plan covers up to 1,000 users.

## Product

- [Pricing](https://www.example.com/pricing): Plans, limits and what each plan includes
- [Integrations](https://www.example.com/integrations): Supported data sources and destinations

## Company

- [Security](https://www.example.com/security): Data hosting regions, retention and certifications

Note what makes them consistent. Every page in llms.txt is in the sitemap. Nothing in either file sits under a disallowed path. The lastmod dates are real edit times, not the time of the last deploy. And the summary in llms.txt repeats facts that the pricing page states in its own words.

Where the three files contradict each other

These are common conflicts, roughly in order of damage:

• A staging robots.txt reaches production. Disallow: / for every user agent is common on staging sites. Shipped to production, it blocks every compliant crawler at once.
• robots.txt returns an error. A 5xx response means "assume complete disallow." A 404 is safer, because the RFC lets crawlers treat a missing file as permission to fetch everything.
• The sitemap lists blocked or redirected URLs. A URL that robots.txt forbids, or that redirects elsewhere, wastes crawl attention. List only final, canonical URLs that return 200.
• lastmod is the build time. If every URL shows the same timestamp after each deploy, search engines learn to ignore the field. The sitemap protocol says the date must reflect when "the linked page was last modified, not when the sitemap is generated."
• llms.txt links to pages crawlers cannot read. Links behind a login, under a disallowed path or rendered only by JavaScript lead agents to nothing useful.
• llms.txt says something the pages don't. If the file claims "free plan available" and the pricing page no longer has one, every agent that checks the page will notice the conflict.
• Training and search are lumped together. Blocking every AI user agent to stop training also removes you from AI search answers. Decide each use separately.

A 20-minute consistency check

curl -sI https://www.example.com/robots.txt
curl -sI https://www.example.com/sitemap.xml
curl -sI https://www.example.com/llms.txt

Where sources disagree

Google and Bing agree that priority and changefreq do nothing and that an accurate lastmod helps, but Bing ties sitemaps to AI answers more explicitly than Google does. On llms.txt the split is sharper: the proposal's author describes broad use by coding agents, while Google says Search ignores the file. Both can be true. A search index does not need llms.txt, while an agent already working on your site may use it. Our agent-ready website checklist shows where these files fit among the other things that make a site readable for agents.

If you want a starting point for the third file, the llms.txt generator drafts one from your existing pages.

None of these files will make an AI system describe you well on its own. They decide whether machines may read your pages, whether they find them, and whether they read the right ones first. Keep the three in agreement, keep the facts in the HTML, and the pages themselves can do the rest.