You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have read the Search Console crawl stats next to the invoice, watched referrals from AI assistants appear in the source report, and blocked a training crawler only to discover which product tokens it did and did not share with the search index. You treat crawler policy as a portfolio decision made from measured returns, revisited on a schedule, rather than a stance.

## Key Points

- **`Google-Extended`** stops fetched pages being used for Gemini training and grounding. It changes nothing about crawl volume or Search.
- **Disallowing `ClaudeBot`** stops Anthropic's training crawl; Anthropic documents `Claude-SearchBot` and `Claude-User` as separate tokens. Same structure, same consequence.
- **`PerplexityBot`** governs Perplexity's index; Perplexity documents that `Perplexity-User` may not honour robots.txt because a person requested the page.
- **A `Disallow` does not de-index**: a URL that is linked elsewhere can remain in the index without content. To remove a page, allow the crawl and serve `noindex`.
2. Create a custom channel group with an "AI assistants" channel matched by regular expression on source so the segment persists in standard reports.
4. Search Console does not split AI Overview impressions and clicks from Web search; treat Google's AI features as part of organic and do not try to isolate them from that report.
5. Record the monthly numbers: organic clicks, AI-assistant sessions, AI-assistant engaged sessions, canary hits. These four numbers are the return side of the ledger.
- `rel="canonical"` on every page, self-referencing, absolute, consistent with the sitemap and with internal links.
- Sitemap `lastmod` that is accurate; Google uses it when it is consistently truthful and ignores it when it is not. Drop `changefreq` and `priority`; they are ignored.
- JSON-LD appropriate to the content (`Article`, `TechArticle`, `SoftwareApplication`, `Dataset`, `FAQPage` where genuinely a FAQ) with `author`, `datePublished`, `dateModified` and `url`.
- Plain-text facts in the HTML: the definitions, numbers and names people ask about, not in images and not behind a click.
- Stable URLs; a 301 chain loses a citation more often than it loses a ranking.

## Quick Example

```text
# GA4 custom channel group, channel "AI assistants": Session source matches regex
chatgpt\.com|perplexity\.ai|claude\.ai|copilot\.microsoft\.com|gemini\.google\.com
```

```html
<!-- Limit what Google quotes in results and AI features without leaving the index -->
<meta name="robots" content="max-snippet:160, max-image-preview:large">
<section data-nosnippet>Text that may be indexed but must not be quoted in a snippet.</section>
```

SEO and AI Search Visibility Trade-offs

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have read the Search Console crawl stats next to the invoice, watched referrals from AI assistants appear in the source report, and blocked a training crawler only to discover which product tokens it did and did not share with the search index. You treat crawler policy as a portfolio decision made from measured returns, revisited on a schedule, rather than a stance.

Core Principle: Admit Crawlers by What They Return

Every crawler costs you requests, CPU and egress. The question is what comes back. Four classes, four different answers:

Class	Examples (tokens)	What you get back	What it costs you	Default for a small content site
Search index	Googlebot, bingbot, Applebot, DuckDuckBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot	Clicks, impressions, brand queries	Crawl proportional to site size and change rate; modest	Admit, verify, never challenge
AI assistant and user-triggered fetchers	ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher	Citations, referral clicks, being the source a person is shown	One fetch per user question; tiny	Admit unless referrals measure zero and copying is the concern
Training crawlers	GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent, Amazonbot; control tokens Google-Extended, Applebot-Extended	No direct traffic; an unmeasurable presence in model knowledge	Full-site crawls, sometimes aggressive; several publishers reported heavy load from training crawlers in 2024	Decide on principle and bandwidth; measurable return is nil
Undeclared scrapers	Stock-browser headless clients, curl loops, proxy pools	Nothing; copies of your content and poisoned analytics	Whatever your cheapest path costs per page, times their volume	Rate limit, gate, detect; not a robots.txt matter

The measured case in this pack: one undeclared headless crawler cost under $2 a month in hosting but was 87% of "users" and had copied 80% of the catalogue three times. Declared crawlers on the same site fetched a fraction of that and produced all of its organic traffic. The policy fight is almost never with the crawlers that obey policy.

What Blocking Actually Does

• Disallowing Googlebot removes you from Google Search, Discover, AI Overviews and AI Mode together; they share the crawl. There is no token that opts out of AI Overviews alone. The only documented lever on how much of your text appears in AI features without leaving the index is the snippet controls: nosnippet, max-snippet:N, and data-nosnippet on elements.
• Google-Extended stops fetched pages being used for Gemini training and grounding. It changes nothing about crawl volume or Search.
• Disallowing GPTBot stops OpenAI training crawls; per OpenAI's documentation it does not affect OAI-SearchBot (ChatGPT search index) or ChatGPT-User (fetch on a user's behalf). Disallowing OAI-SearchBot removes you from ChatGPT search results and with them the citations.
• Disallowing ClaudeBot stops Anthropic's training crawl; Anthropic documents Claude-SearchBot and Claude-User as separate tokens. Same structure, same consequence.
• PerplexityBot governs Perplexity's index; Perplexity documents that Perplexity-User may not honour robots.txt because a person requested the page.
• Blocking at the edge (a 403, a challenge) is stronger than robots.txt and less precise: Search Console reports the errors, the URLs are dropped after repeated failures, and no vendor distinguishes "you challenged me" from "your site is down".
• A Disallow does not de-index: a URL that is linked elsewhere can remain in the index without content. To remove a page, allow the crawl and serve noindex.

Crawl Budget and Crawl Stats

Crawl budget matters for sites with around a million URLs, or tens of thousands of pages that change daily, not for a catalogue of a few thousand pages; Google's own crawl-budget documentation says as much. What matters at small scale is reading Search Console's Crawl stats report (Settings, Crawl stats): total crawl requests per day, total download size, average response time, breakdown by response code, by file type, by purpose (discovery versus refresh) and by Googlebot type, and host status. Multiply requests by average size and you have Google's monthly egress on your site to compare with the invoice; typically it is a rounding error next to a single undeclared scraper. A rising average response time on that report is the earliest sign that something else is consuming your instance.

Bing Webmaster Tools has an equivalent crawl report. For AI-search and assistant fetchers there is no dashboard; the access logs are the only source, so log the User-Agent and count by token.

Worked Example: One Site's Crawl Ledger

Per-token request counts come from the logs. Only the undeclared row below is the measured case from this pack's background; the declared rows are illustrative of the shape a few-thousand-page site sees. Build your own before deciding anything.

SELECT
  CASE
    WHEN http_request.user_agent LIKE '%Googlebot%'     THEN 'Googlebot'
    WHEN http_request.user_agent LIKE '%bingbot%'       THEN 'bingbot'
    WHEN http_request.user_agent LIKE '%OAI-SearchBot%' THEN 'OAI-SearchBot'
    WHEN http_request.user_agent LIKE '%ChatGPT-User%'  THEN 'ChatGPT-User'
    WHEN http_request.user_agent LIKE '%GPTBot%'        THEN 'GPTBot'
    WHEN http_request.user_agent LIKE '%Claude-User%'   THEN 'Claude-User'
    WHEN http_request.user_agent LIKE '%ClaudeBot%'     THEN 'ClaudeBot'
    WHEN http_request.user_agent LIKE '%PerplexityBot%' THEN 'PerplexityBot'
    WHEN http_request.user_agent LIKE '%CCBot%'         THEN 'CCBot'
    WHEN http_request.user_agent LIKE '%Bytespider%'    THEN 'Bytespider'
    ELSE 'undeclared or other'
  END AS token,
  COUNT(*) AS requests,
  COUNTIF(http_request.request_url LIKE '%/skills/%') AS html_pages,
  ROUND(SUM(http_request.response_size) / 1e6, 1) AS megabytes
FROM `project.region.bucket._AllLogs`
WHERE resource.type = 'cloud_run_revision'
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY 1 ORDER BY requests DESC;

Token	Requests / 30 days	Data served	Return / 30 days	Decision
Googlebot	~6,000	~150 MB	~1,900 organic clicks and every impression	Admit, verify, exempt from limits
bingbot	~2,500	~60 MB	~140 clicks	Admit, verify, exempt
OAI-SearchBot and ChatGPT-User	~900	~20 MB	~35 sessions tagged utm_source=chatgpt.com, most engaged	Admit; re-measure quarterly
PerplexityBot and Perplexity-User	~400	~10 MB	~8 referral sessions	Admit; re-measure quarterly
GPTBot	~4,000	~100 MB	none measurable	Disallow by token, or allow on principle
ClaudeBot, CCBot, Bytespider	~3,000 combined	~70 MB	none measurable	Same
Undeclared headless Chrome	15,000 HTML plus ~300,000 assets	~5 GB	none; 87% of reported users; 80% of the catalogue copied three times	Rate limit, delayed beacon, log; not a robots.txt matter

Read across the rows: the search rows cost a few hundred megabytes and return every organic click; the assistant rows cost almost nothing and return a small, growing number; the training rows cost about as much as search and return nothing you can count; the undeclared row costs more than every declared token combined and returns less than nothing. Only the last row moves the invoice, and only the first two rows move revenue.

Measuring Return from AI Surfaces

1. In GA4 (or Plausible, Fathom), filter sessionSource for the assistant domains: chatgpt.com, perplexity.ai, claude.ai, copilot.microsoft.com, gemini.google.com, and their mobile-app variants. ChatGPT appends utm_source=chatgpt.com to outbound links, which lands in sessionSource and sessionMedium directly; other assistants rely on the Referer header, which apps often strip, so part of this traffic appears as (direct).
2. Create a custom channel group with an "AI assistants" channel matched by regular expression on source so the segment persists in standard reports.
3. Count citations without clicks by searching your canary sentences and brand in each assistant monthly; a citation that is read inside the answer without a click still has value that referrals understate.
4. Search Console does not split AI Overview impressions and clicks from Web search; treat Google's AI features as part of organic and do not try to isolate them from that report.
5. Record the monthly numbers: organic clicks, AI-assistant sessions, AI-assistant engaged sessions, canary hits. These four numbers are the return side of the ledger.

# GA4 custom channel group, channel "AI assistants": Session source matches regex
chatgpt\.com|perplexity\.ai|claude\.ai|copilot\.microsoft\.com|gemini\.google\.com

<!-- Limit what Google quotes in results and AI features without leaving the index -->
<meta name="robots" content="max-snippet:160, max-image-preview:large">
<section data-nosnippet>Text that may be indexed but must not be quoted in a snippet.</section>

Signals That Make the Return Land on You

Being crawled is not enough; the crawler has to prefer your page over copies of it.

• rel="canonical" on every page, self-referencing, absolute, consistent with the sitemap and with internal links.
• Sitemap lastmod that is accurate; Google uses it when it is consistently truthful and ignores it when it is not. Drop changefreq and priority; they are ignored.
• JSON-LD appropriate to the content (Article, TechArticle, SoftwareApplication, Dataset, FAQPage where genuinely a FAQ) with author, datePublished, dateModified and url.
• Plain-text facts in the HTML: the definitions, numbers and names people ask about, not in images and not behind a click.
• Stable URLs; a 301 chain loses a citation more often than it loses a ranking.
• Last-Modified and ETag headers so polite crawlers refresh with 304s instead of full fetches, which cuts the cost of the crawlers you do want.

A Policy Matrix Small Sites Can Adopt

Situation	Search index	AI search and assistant fetchers	Training crawlers	Undeclared scrapers
Traffic-dependent content site (ads, sponsorship, reach)	Allow, verify, exempt from limits	Allow; measure referrals quarterly	Disallow by token; nothing measurable comes back	Rate limit HTML navigations; delayed beacon; detect copies
Product docs and marketing	Allow	Allow; assistants answering "how do I" with your docs is the product	Allow or disallow on principle; either is defensible	Rate limit; keep docs fully open
Licensed or paid content	Allow with paywalled-content markup	Allow the preview; gate the rest server-side	Disallow	Gate server-side; API with keys
Personal site, no monetisation	Allow	Allow	Personal choice	Rate limit only if the bill or analytics say so
Site under active copying by a competitor	Allow	Allow	Disallow	Gate, canary, rate limit, then block on evidence

The matrix says the same thing in every row for the first column, because search is the only class whose return is unambiguous, and it says "measure" for the second column because assistant referrals were near zero for most small sites in 2024 and growing through 2025 and 2026; the number on your own site decides.

Procedure: Setting and Revisiting the Policy

1. Pull 90 days of organic clicks, AI-assistant sessions and canary hits. Pull the crawl stats report and the per-token request counts from the logs.
2. Cost each declared token's crawl (requests times bytes) and compare it to its return column.
3. Choose the row of the matrix that describes the site and write the robots.txt for it (three postures are in the robots.txt skill).
4. Handle undeclared traffic separately: rate limits, beacon delay, gating and detection are not policy, they are engineering.
5. Record the decision with the numbers and the date in the repository next to robots.txt.
6. Quarterly: re-read each vendor's crawler documentation for new or renamed tokens (new tokens have appeared every year since 2023: Google-Extended and GPTBot in 2023, OAI-SearchBot, Applebot-Extended and meta-externalagent in 2024, the Anthropic search and user tokens and Perplexity-User in 2025); grep the logs for tokens you have never seen; re-pull the four return numbers; adjust the file.
7. After any change, watch Search Console coverage and crawl stats for two weeks and the AI-assistant channel for a quarter before judging it.

Checklist

• Return measured per class: organic clicks, AI-assistant sessions, citations, nothing.
• Cost measured per class from logs and crawl stats.
• Search crawlers verified and exempt from every limit and challenge.
• Snippet controls chosen deliberately if AI feature display is a concern; Googlebot not blocked to achieve it.
• Training tokens decided on principle with the decision recorded.
• Undeclared traffic handled by engineering, not by robots.txt.
• Canonicals, sitemap lastmod, structured data and cache validators in place.
• Policy review scheduled quarterly with the vendor token pages bookmarked.

Common Mistakes

• Blocking Googlebot or challenging it to keep text out of AI Overviews.
• Blocking GPTBot and expecting ChatGPT search citations to stop, or blocking OAI-SearchBot and expecting them to continue.
• Reading (direct) traffic as a bot when a share of it is assistant apps stripping referrers.
• Treating crawl budget as a small-site problem.
• Fighting the training crawlers in robots.txt while an undeclared headless browser, which never read the file, does all the damage.
• Setting a policy once in 2024 and never re-reading the token list.

Limits

Returns from AI surfaces are still poorly measured: citations without clicks are invisible to analytics, referrers are stripped by apps, and no vendor gives a publisher-side report comparable to Search Console. The matrix above is a starting position to be overridden by your own numbers, not a rule. And nothing in this skill touches the undeclared scraper that costs the most and returns the least; that is handled by rate limiting, content gating, access-log forensics and edge controls in the other skills of this pack.