You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have crawled other people's sites and found their entire catalogue in one JSON file, and you have found your own in someone else's product. You start every gating conversation with an inventory of what the site actually ships, because a gate placed anywhere except the cheapest path is decoration, and you weigh every gate against the indexing and citation it will cost.

## Key Points

- "Hydrate then fetch on click": the click calls an endpoint. The endpoint is now the thing to protect; if it accepts any GET with no session, key or token, the gate moved and did not close.
- Obfuscation (right-click disabling, text rendered into canvas, font glyph shuffling): breaks accessibility and search, and costs a scraper one OCR pass.
- `Copy` event interception: irrelevant to any client that is not a person.
- Decide the preview length by content type, not a global percentage: a reference page needs the definition and the first example; an article needs its lede.
- Keep the preview above the fold in the HTML, not injected later.
- Do not gate the pages you want ranking on competitive queries; gate the long tail, or gate nothing and rely on rate limits.
- Record every full-text grant with the key, IP and User-Agent so a copy can be traced to a grant.
- **Per-recipient variants**: for API and gated deliveries, vary a detail per key (sentence order in a list, a synonym choice, a numbering style) and keep the mapping. A copy identifies the key.
- Text not delivered in the HTML the crawler receives is not indexed by that crawler. Google renders JavaScript with delay and budget; content behind a click is not seen.
- AI search and assistant fetchers cite what they can fetch. A preview that omits the fact the user asked about produces no citation, or a citation to a competitor's copy.
- A preview must still contain the entities and facts that queries target, in plain text, with the canonical URL and structured data intact, so that the indexed page is yours rather than a copier's.
1. Run the inventory table against the site; list every path that ships full text and every dataset under `/public`.

Content Exposure and Preview Gating

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have crawled other people's sites and found their entire catalogue in one JSON file, and you have found your own in someone else's product. You start every gating conversation with an inventory of what the site actually ships, because a gate placed anywhere except the cheapest path is decoration, and you weigh every gate against the indexing and citation it will cost.

Core Principle: You Ship Whatever the Cheapest Path Ships

A crawler's cost to copy your content is set by the cheapest route to the text, not by the route you designed for people. A site that renders a "show more" button over server-rendered HTML ships the full text to every client on every page; a site with /data/catalogue.json ships the entire corpus in one request and can be copied in a second without a headless browser. The measured crawler in this pack's background spent 90 days and a headless Chrome reading 80% of a catalogue three times at 330 KB a visit, which means the site had no cheaper route; that is the only reason it cost the crawler anything at all.

So the sequence is fixed: inventory every path, decide what should be free and indexable and what should not, put the gate on the server for the parts that should not, keep the free part large enough to be indexed and cited, and detect copies rather than pretend to prevent them.

Inventory: Where Full Text Ships

Path	What a client gets	Check
Server-rendered HTML	Full text in the document body, plus <meta name="description"> and JSON-LD articleBody if you emit it	curl -s URL | wc -c and read it
React Server Component payload	The same props and text as Flight data, fetched with RSC: 1 on client navigation; anyone can send that header	curl -s -H 'RSC: 1' URL
Pages Router __NEXT_DATA__	All getStaticProps / getServerSideProps data as inline JSON in the HTML	View source, search for __NEXT_DATA__
Pages Router data routes	/_next/data/<buildId>/<route>.json returns props without the HTML	Watch the network panel on a client navigation
Prerendered static routes	Same content, cheaper for you and for the copier; often served from a CDN with no logging	Look for x-vercel-cache or age headers
JSON, CSV or Markdown under /public	Whole datasets in one request	ls public/ and read your build script
Sitemaps	A complete, ordered list of every URL: the crawl plan	curl -s URL/sitemap.xml | grep -c '<loc>'
Public APIs and search endpoints	?q=a or an empty query that returns everything; pagination with no upper bound	Read the handlers
RSS and Atom	Full-content feeds hand over the body of every item	<content:encoded> versus <description>
MCP servers, SDKs, CLIs, npm packages	A documented, rate-limited bulk route, which is good, if it has quotas	Read the quota code
Open-source repos	The content itself, if the content is in git	Check the licence file
Third-party copies	Common Crawl, the Wayback Machine, search engine caches	Search a canary sentence
OG images and screenshots	Rendered text in images	Look at what the image generator draws

A gate anywhere is meaningful only after this table has no unintended "full text" row. The inventory takes ten minutes:

URL=https://example.com/skills/example-page
curl -s "$URL" | wc -c                                              # HTML bytes: full text or preview?
curl -s -H 'RSC: 1' "$URL" | wc -c                                  # RSC payload; must not exceed what the HTML shows
curl -s "$URL" | grep -c '__NEXT_DATA__'                            # Pages Router inline props present?
curl -s https://example.com/sitemap.xml | grep -c '<loc>'           # size of the crawl plan you publish
curl -s 'https://example.com/api/search?q=a&limit=100000' | wc -c   # does a bulk route exist, and is it bounded?
curl -s https://example.com/feed.xml | grep -c '<content:encoded>'  # full-content feed?
find public -type f \( -name '*.json' -o -name '*.csv' -o -name '*.md' \) -size +100k   # datasets shipped verbatim

Why Client-Side Gating Is Theatre

• max-height with a fade, display: none, blur, or a "show 30%" React state: the full text is in the HTML or in the RSC payload. Every client that reads HTML has it, including search engines, which index what they receive.
• "Hydrate then fetch on click": the click calls an endpoint. The endpoint is now the thing to protect; if it accepts any GET with no session, key or token, the gate moved and did not close.
• Obfuscation (right-click disabling, text rendered into canvas, font glyph shuffling): breaks accessibility and search, and costs a scraper one OCR pass.
• Copy event interception: irrelevant to any client that is not a person.

If the goal is to make copying cost something, the text that should cost something must not leave the server without a check.

Server-Side Preview Gating

Render, for every client, a preview that is worth indexing: title, a summary that carries the key facts, the first section or the first N% of the body, structured metadata, and a clear statement of what the full version contains. Serve the rest only on a server-side condition: a session cookie issued after a real interaction, an API key with a quota, a signed token from a challenge widget, or simply the origin rate limit passing. The condition runs before render, so the RSC payload and the HTML agree.

Google's documented mechanism for indexing text that most clients cannot see is the paywalled-content structured data: isAccessibleForFree: false plus hasPart entries with cssSelector naming the gated sections, and the full text served to verified Googlebot. Google states this is not cloaking when the markup is present and accurate; serving different text to Googlebot without it is cloaking and risks manual action. "Flexible sampling" (metering or a lead-in) is Google's stated preference for how much to show. Other search and AI-search crawlers have no equivalent contract: they index what they receive, so the preview you show everyone is what they cite.

Implementation notes:

• Decide the preview length by content type, not a global percentage: a reference page needs the definition and the first example; an article needs its lede.
• Keep the preview above the fold in the HTML, not injected later.
• Do not gate the pages you want ranking on competitive queries; gate the long tail, or gate nothing and rely on rate limits.
• Record every full-text grant with the key, IP and User-Agent so a copy can be traced to a grant.

// app/skills/[slug]/page.tsx — the decision is made before render, so HTML and RSC payload agree
export default async function SkillPage({ params }: { params: { slug: string } }) {
  const doc = await loadSkill(params.slug);
  const grant = await fullTextGrant();          // session cookie, API key, challenge token, or rate-limit pass
  if (grant) await logGrant({ slug: params.slug, grant });   // key or IP plus User-Agent, for tracing copies
  return (
    <Article
      title={doc.title}
      summary={doc.summary}                     // always present: the indexable, citable part
      body={grant ? doc.body : doc.preview}     // preview = first section, truncated on the server
      gated={!grant}
    />
  );
}

{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "headline": "Example skill page",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".gated-body"
  }
}

API-First Access

The crawler that wanted your corpus will take an API if the API is cheaper than maintaining a headless browser. Publish one: keys, per-key quotas (daily and per-minute), a free tier sized for individual use, terms that say what redistribution is permitted, and a contact for bulk licensing. Then the cost of the undeclared route can be raised (rate limits, preview gating) without cutting off the legitimate use, and every full-text access from the API is attributable to a key.

Quota design mirrors page rate limiting: a daily bucket for volume, a per-minute bucket for bursts, 429 with Retry-After, and a log line per call.

Detecting Copies: Canaries and Watermarks

Prevention fails eventually; detection is cheap.

• Canary sentences: one natural, unique sentence per document, or per section, that would not occur anywhere else. Search for exact matches in search engines monthly and ask AI answer engines about the sentence. A hit tells you which document was copied and, if the sentence varies per delivery path, which path.
• Per-recipient variants: for API and gated deliveries, vary a detail per key (sentence order in a list, a synonym choice, a numbering style) and keep the mapping. A copy identifies the key.
• Honeypot URLs: a path listed in robots.txt as Disallow and referenced nowhere a person would click. Any hit is an undeclared crawler that read robots.txt and ignored it, or a scraper following every URL. Do not hide the link in the visible page; hidden links are a search spam signal and confuse assistive technology. A sitemap entry or a <link rel="alternate"> that no human follows is enough.
• What not to do: zero-width characters (stripped by any normaliser, flagged by some scanners), homoglyphs (break search, accessibility and copy-paste for humans), invisible text (a Google spam policy violation).

Detection gives you evidence for a takedown request, a licensing conversation, or a block rule with a reason on it.

The Visibility Cost

• Text not delivered in the HTML the crawler receives is not indexed by that crawler. Google renders JavaScript with delay and budget; content behind a click is not seen.
• AI search and assistant fetchers cite what they can fetch. A preview that omits the fact the user asked about produces no citation, or a citation to a competitor's copy.
• Snippet controls (<meta name="robots" content="max-snippet:N">, nosnippet, data-nosnippet on an element) limit what Google shows in results and in its AI features without hiding the text from the index. This is the documented way to show less in an AI Overview while staying indexed; it does not stop the text being read.
• A preview must still contain the entities and facts that queries target, in plain text, with the canonical URL and structured data intact, so that the indexed page is yours rather than a copier's.

Procedure

1. Run the inventory table against the site; list every path that ships full text and every dataset under /public.
2. Remove or gate the bulk routes first (JSON catalogues, unbounded search, full-content feeds): those set the copying price.
3. Decide per content type what is free, what is preview, and what is full-only.
4. Implement gating server-side, before render, with the grant condition and the grant log.
5. Add paywalled-content structured data if Googlebot should still see the full text; verify with the Rich Results Test and Search Console's URL inspection.
6. Publish the API with keys and quotas; link it from the preview.
7. Seed canaries and schedule a monthly search.
8. Measure Search Console impressions and AI-surface referrals for four weeks after the change; roll back the gate on any page where the loss exceeds what the copies cost you.

Checklist

• No bulk route ships the corpus without a key and quota.
• RSC payload and HTML deliver the same gated view.
• Preview carries title, summary with key facts, first section, canonical URL and structured data.
• Full text grants are logged with key, IP and User-Agent.
• Paywalled-content markup present if Googlebot sees more than others.
• Canaries seeded and searched monthly.
• Search Console and referral impact reviewed four weeks after gating.

Common Mistakes

• Gating the HTML and forgetting the RSC payload, the data route, or the feed.
• Leaving the site's own JSON index under /public because the front end uses it.
• Showing 30% client-side and calling it protection.
• Serving Googlebot the full page without the structured data and getting a cloaking action.
• Gating the pages that drive search traffic.
• Using invisible text or hidden links as honeypots.
• Assuming a copy is prevented rather than detected.

Limits

This skill controls what the server hands out and how copies are found. It cannot stop a licensed or determined party from retyping a preview, and it does not decide who deserves the full text; that is a business and visibility question the trade-off skill in this pack frames. On a site inside every free tier, where the harm is analytics contamination rather than copying, gating is usually the wrong first move: a delayed beacon and origin rate limits fix the measurement problem without costing a single indexed sentence.