What is robots.txt? Definition and examples

robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may fetch, following the Robots Exclusion Protocol (RFC 9309).

How it works

The convention dates back to the 1990s and was published as an IETF standards-track document, RFC 9309, in September 2022 (RFC 9309). A crawler requests /robots.txt before crawling a site, finds the group of rules that matches its user agent, and skips disallowed paths.

The file is made of groups:

• User-agent: names the crawler the group applies to (* means any crawler without a more specific group).
• Disallow: and Allow: list path prefixes. When rules conflict, the most specific (longest) match wins.
• Sitemap: points to your XML sitemap and can appear anywhere in the file.

A few details from the RFC that trip people up: each host (including each subdomain) needs its own file; crawlers may cache the file but should not use a cached copy for more than 24 hours unless the file is unreachable; and parsers must handle at least 500 kibibytes, so very long files risk being cut off.

What it does not do

robots.txt is a request, not a lock. The RFC says the rules "are not a form of access authorization," and Google says robots.txt "is not a mechanism for keeping a web page out of Google": a disallowed page can still be indexed if other sites link to it. To keep a page out of search results, use a noindex directive or password protection instead (Google Search Central). And because compliance is voluntary, badly behaved bots may ignore the file entirely.

Why it matters for your website

robots.txt is now also where you state your policy on AI crawlers. Each AI company publishes the user agent tokens it honors, often separating training, search indexing and user-requested fetches. OpenAI, for example, notes that each of its settings "is independent of the others", so a site can allow search crawling while disallowing training (OpenAI).

A wrong line can quietly remove you from search or AI answers, so review the file whenever you change it, and remember that a CDN or firewall may block bots before they ever read it.

Example

# Everyone: crawl the site, but not internal search results or the app
User-agent: *
Disallow: /search
Disallow: /app/

# OpenAI: allow ChatGPT search, opt out of model training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Related terms

• AI crawler: the AI bots and their user agent tokens.
• llms.txt: a proposed companion file that points AI tools to key pages.
• Structured data: what crawlers can read inside your pages.
• Zero-click search: why visibility in answers matters even without visits.