You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have written robots.txt files that crawlers honoured and files that they ignored, and you have written crawlers that read them and, once, one that did not. You treat robots.txt as a published policy that well-behaved parties follow and that badly behaved parties tell you nothing about, and you never confuse writing a rule with enforcing one.

## Key Points

- **Case**: paths are case-sensitive. `Disallow: /Admin` does not block `/admin`.
- **`Sitemap:`** is independent of groups and may appear anywhere; it must be an absolute URL.
- **Size**: Google reads only the first 500 KiB. Keep the file small.
- **Comments** start with `#`. Blank lines separate groups.
- **IETF AI Preferences (aipref) work** is drafting a vocabulary for expressing training and inference preferences in robots.txt and HTTP headers. Track it; do not build on it yet.
1. Write the file with explicit groups per posture; keep the `*` group last and complete.
2. Serve it with `Content-Type: text/plain; charset=utf-8` and status `200`; confirm it is not behind a challenge page or an auth redirect.
3. Test it:
4. Evaluate a URL against a token with Google's open-source parser (`github.com/google/robotstxt`, builds a `robots_main` binary):
7. Watch for new tokens in the logs each quarter and re-read the vendor crawler pages; new tokens have appeared every year since 2023.
- It cannot remove a URL from an index. A disallowed URL that is linked from elsewhere can still be indexed without content. To de-index, allow the crawl and serve `noindex`.
- It cannot reduce crawl load from `Google-Extended` or `Applebot-Extended`; those are usage flags.

## Quick Example

```text
# Posture A: maximum reach. Everyone welcome, admin paths excluded.
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Sitemap: https://example.com/sitemap.xml
```

```bash
curl -sS -D - https://example.com/robots.txt -o /dev/null | head -5   # status and content-type
curl -sS https://example.com/robots.txt                                # the body you think you deployed
```

robots.txt and AI Crawler Policy

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have written robots.txt files that crawlers honoured and files that they ignored, and you have written crawlers that read them and, once, one that did not. You treat robots.txt as a published policy that well-behaved parties follow and that badly behaved parties tell you nothing about, and you never confuse writing a rule with enforcing one.

Core Principle: robots.txt Is a Notice, Not a Lock

RFC 9309 (the Robots Exclusion Protocol, standardised in 2022) describes a plain-text file at /robots.txt that crawlers fetch and voluntarily obey. Nothing in HTTP enforces it. Its value is that the crawlers that account for your visibility (search engines, AI search indexes, assistant fetchers) do obey it, and they publish exactly which product token they check. So robots.txt is the cheapest, most precise control you have over the traffic that matters, and it is no control at all over the traffic that does not care. Both halves of that sentence are true and you should say both to anyone who asks whether "blocking bots in robots.txt" will fix their bill.

Syntax and Semantics That Actually Matter

• Location: exactly /robots.txt at the root of the scheme, host and port. https://example.com/robots.txt does not cover http://example.com or https://www.example.com. Subdirectory files are ignored.
• Groups: one or more User-agent: lines followed by Allow: / Disallow: lines. A crawler picks the group whose product token matches it most specifically; if none matches, it uses the User-agent: * group. Groups do not inherit from *: a crawler with its own group ignores every rule in the * group.
• Precedence: for a given URL, the rule with the longest matching path wins. On a tie, Google and most large engines let Allow win; the RFC says use the least restrictive rule, which is the same outcome.
• Wildcards: * matches any sequence and $ anchors the end of a path. RFC 9309 defines both as special characters that conforming parsers must support, and Google, Bing, Apple, OpenAI, Anthropic and the major crawlers do; older and hand-written parsers often do not, so keep critical rules literal where you can.
• Case: paths are case-sensitive. Disallow: /Admin does not block /admin.
• Sitemap: is independent of groups and may appear anywhere; it must be an absolute URL.
• Crawl-delay: is not in the RFC. Bing honours it (as a minimum spacing between requests). Google ignores it entirely and retired its crawl-rate limiter in Search Console in 2024; if Googlebot is too fast, return 429 or 503 briefly and it slows down. Yandex honoured it historically. Most AI crawlers document no support; a few say they honour it. Treat it as a request, not a control.
• Size: Google reads only the first 500 KiB. Keep the file small.
• HTTP status semantics: 200 is parsed. 4xx (including 404) is treated as "no restrictions" by Google and most engines. 5xx is treated by Google as a temporary full disallow; if it persists for around 30 days Google falls back to its last cached copy, or to no restrictions if it has none. Never let robots.txt return 5xx when a deploy fails.
• Comments start with #. Blank lines separate groups.

The Tokens That Matter, Grouped by Purpose

Tokens as documented by their vendors at time of writing; vendors add and rename tokens, so re-verify each quarter against the vendor's crawler page.

Search index crawlers (they send you traffic)

Token	Operator	Notes
Googlebot	Google	Web Search, Discover, AI Overviews and AI Mode all use Googlebot's crawl
bingbot	Microsoft	Bing, Copilot's web results, DuckDuckGo results partly, Yahoo
Applebot	Apple	Siri, Spotlight, Safari suggestions
DuckDuckBot	DuckDuckGo	Supplements Bing results
YandexBot, Baiduspider	Yandex, Baidu	Regional; block only if you have no audience there
OAI-SearchBot	OpenAI	Index behind ChatGPT search results; not used for training per OpenAI's documentation
Claude-SearchBot	Anthropic	Index behind Claude's search results; documented as separate from training
PerplexityBot	Perplexity	Perplexity's search index

AI training controls (no direct return traffic)

Token	Operator	Notes
GPTBot	OpenAI	Training crawler; publishes its IP ranges
ClaudeBot	Anthropic	Training crawler
Google-Extended	Google	Not a crawler: a control token that tells Google not to use pages Googlebot fetched for Gemini training and grounding. Does not affect Search, AI Overviews or AI Mode
Applebot-Extended	Apple	Same pattern: a control token; Applebot still crawls for search
CCBot	Common Crawl	Builds the open corpus that many models train on
Bytespider	ByteDance	Training and product crawler; widely reported as aggressive and inconsistent about robots.txt
meta-externalagent	Meta	Training and indexing crawler
Amazonbot	Amazon	Alexa and Amazon services; documented as improving products, which includes model use
Google-CloudVertexBot	Google	Crawls on behalf of a Vertex AI customer who built an agent pointed at your site; rarely worth allowing unless that customer is you

AI assistant and user-triggered fetchers (citations and referrals)

Token	Operator	Notes
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user asks about it or clicks a source
Claude-User	Anthropic	Same pattern for Claude
Perplexity-User	Perplexity	Perplexity documents that user-requested fetches may ignore robots.txt because a person asked
meta-externalfetcher	Meta	User-initiated fetches for Meta AI

Two facts that change decisions: Google-Extended and Applebot-Extended cannot reduce your crawl load because the crawling is done by Googlebot and Applebot regardless; they only change what the fetched pages may be used for. And the search-index and assistant tokens are documented by their vendors as independent of the training tokens, so blocking GPTBot does not remove you from ChatGPT search, and blocking ClaudeBot does not stop Claude-User fetching a page a person asked about.

Three Postures Small Content Sites Actually Adopt

# Posture A: maximum reach. Everyone welcome, admin paths excluded.
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Sitemap: https://example.com/sitemap.xml

# Posture B: search and AI search yes, training no. The common choice.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Google-CloudVertexBot
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Sitemap: https://example.com/sitemap.xml

# Posture C: search only. AI search and assistants excluded too.
# Costs citations from every AI surface; only sensible if you have measured no referrals from them.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalfetcher
Disallow: /
# ... plus the Posture B training group and the * group

A rule that trips people: because groups do not inherit, in Posture B the training crawlers do not receive Disallow: /admin/; they receive Disallow: /, which covers it. But if you write a group for bingbot that only adds a Crawl-delay, you must repeat the /admin/ disallow in that group or Bing will crawl it.

llms.txt and the Newer Signals

• /llms.txt is a 2024 proposal (from the fast.ai / Answer.AI community) for a Markdown file at the site root giving language models a curated overview and links. It is not an access control and no major search or AI vendor has publicly committed to reading it at the time of writing. It costs one static file to publish; expect it to control nothing.
• Content Signals (a 2025 proposal from a large CDN vendor, published at contentsignals.org) extends robots.txt with a Content-Signal: line expressing search, ai-input and ai-train preferences. Adoption by crawlers is voluntary and partial; check current vendor statements before relying on it.
• IETF AI Preferences (aipref) work is drafting a vocabulary for expressing training and inference preferences in robots.txt and HTTP headers. Track it; do not build on it yet.
• Snippet controls (<meta name="robots" content="nosnippet">, max-snippet, data-nosnippet) are honoured by Google, including in its AI features, and are the only Google-documented way to limit how much text appears in AI Overviews without blocking Googlebot.

Procedure: Publish, Verify, Watch

1. Write the file with explicit groups per posture; keep the * group last and complete.
2. Serve it with Content-Type: text/plain; charset=utf-8 and status 200; confirm it is not behind a challenge page or an auth redirect.
3. Test it:

curl -sS -D - https://example.com/robots.txt -o /dev/null | head -5   # status and content-type
curl -sS https://example.com/robots.txt                                # the body you think you deployed

4. Evaluate a URL against a token with Google's open-source parser (github.com/google/robotstxt, builds a robots_main binary):

robots_main robots.txt GPTBot https://example.com/skills/some-page

5. Check Search Console's robots.txt report (Settings, robots.txt) for the version Google last fetched, fetch errors and parse warnings. The interactive tester was retired; the report is what remains.
6. Grep your access logs for robots.txt fetches by User-Agent. A crawler that never fetched it never read it. A crawler that fetched it and then requested disallowed paths is telling you its token is decorative.
7. Watch for new tokens in the logs each quarter and re-read the vendor crawler pages; new tokens have appeared every year since 2023.

What robots.txt Cannot Do

• It cannot stop an undeclared headless browser. The crawler in the measured case sent a stock Chrome User-Agent, never fetched robots.txt in 90 days, and read 80% of the catalogue three times. Only server-side rate limiting, edge rules or content gating touch that traffic.
• It cannot remove a URL from an index. A disallowed URL that is linked from elsewhere can still be indexed without content. To de-index, allow the crawl and serve noindex.
• It cannot reduce crawl load from Google-Extended or Applebot-Extended; those are usage flags.
• It cannot enforce Crawl-delay on crawlers that do not read it, which includes Google.
• It cannot bind user-triggered fetchers that document their own exception.
• It cannot be verified by the requester's User-Agent alone. Enforcement of a token-based policy at the server (blocking GPTBot even if it did not read the file) requires verifying the token by IP range or reverse DNS, otherwise a scraper claiming to be Googlebot inherits your allow rules.

Checklist

• File is at the root of every scheme and host variant that serves content, or those variants redirect.
• 200 status and text/plain; not challenged, not cached with a stale deploy.
• Each group is complete; nothing relies on inheritance from *.
• Sitemap: is absolute and points at a sitemap that lists only indexable URLs.
• Posture is written down with the reason (measured referrals, measured cost) and the date.
• Access logs are checked for crawlers requesting disallowed paths.
• Token list reviewed against vendor documentation within the last quarter.

Common Mistakes

• Blocking Googlebot to stop AI Overviews. It stops Search.
• Writing Disallow: / under User-agent: * on a staging host and copying it to production.
• Expecting robots.txt to change a cloud bill dominated by an undeclared scraper.
• Using Crawl-delay as the rate limit and never implementing one at the server.
• Adding a group for one crawler to set one directive and forgetting that the group now replaces * for that crawler.
• Treating llms.txt as a control.

Limits

This skill covers the published policy layer. Deciding which posture is right depends on measured referrals and citations, which the visibility trade-off skill covers; enforcing it against clients that ignore the file is the job of rate limiting, edge rules and access-log forensics. Vendor token lists and behaviours change; every table above is a snapshot to be re-verified against the operator's own crawler documentation before you rely on it.