You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have shipped rate limits that stopped a crawler cold and rate limits that stopped nothing because every instance kept its own counter, and you have once locked out an entire university behind one NAT address. You size limits from the measured distribution of real visitors and the measured pattern of the abuser, and you test them against a production build before they meet a real browser.

## Key Points

- **IPv6 /64**: a single subscriber usually has a /64 (or /56); keying on the full address lets a client rotate through 2^64 addresses. Truncate.
- **IPv4 /24**: catches small pools and cloud tenants, and also catches a whole ISP's carrier-grade NAT block. Use only as a second, larger bucket (for example 10× the per-IP limit).
- **Authenticated user**: unlimited or very high for signed-in accounts you can suspend.
1. From request logs, count top-level document navigations per IP per hour and per day over 30 days, excluding known crawlers and your own monitoring.
2. Split by whether the IP had an engaged session (or two or more pages). Take percentiles of the engaged population: `p99` and `p99.9` per hour and per day.
3. Identify your shared-IP outliers: the IPs with the most distinct cookies or User-Agents. Their volume is many humans; your per-IP limit must sit above their daily peak or use cookie keying.
4. Profile the abuser the same way: pages per hour, pages per day, inter-request spacing.
6. Write the four numbers (human p99.9 hour and day, abuser hour and day) next to the limits in the code. The next person will need them.
- Googlebot: published ranges at `developers.google.com/search/apis/ipranges/googlebot.json`, or reverse DNS to `googlebot.com` / `google.com` followed by a forward lookup that returns the same IP.
- bingbot: published ranges at `www.bing.com/toolbox/bingbot.json`, or reverse DNS to `search.msn.com` with forward confirmation.
- Applebot: reverse DNS to `applebot.apple.com` with forward confirmation.
- OpenAI, and other vendors that publish JSON range lists: fetch the list on a schedule, cache it, match by CIDR.

## Quick Example

```bash
for i in $(seq 1 70); do
  curl -s -o /dev/null -w '%{http_code} ' -H 'Sec-Fetch-Dest: document' \
    -H 'User-Agent: test-limit/1.0' http://localhost:3000/skills/example-page
done; echo
```

Web Rate Limiting Design

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have shipped rate limits that stopped a crawler cold and rate limits that stopped nothing because every instance kept its own counter, and you have once locked out an entire university behind one NAT address. You size limits from the measured distribution of real visitors and the measured pattern of the abuser, and you test them against a production build before they meet a real browser.

Core Principle: Limit Navigations, Not Requests

An API rate limit counts calls. A page rate limit must count something a person does: navigating to a page. A single page load on a modern site is one HTML request plus twenty asset requests plus, in an app-router framework, prefetches for every link on screen and a server-component fetch for every client-side navigation. Count all of those and a human reading three pages hits an "abusive" number in a minute. Count only top-level document navigations and the numbers become human-sized, comparable across visitors, and comparable against the crawler.

The second principle: the limit has two jobs, and one bucket cannot do both. A burst limit (per hour) stops a fast scraper. A volume limit (per day) stops the slow, steady crawler that reads 500 pages a day at 21 an hour, which is under any hourly cap generous enough for a person. The measured crawler in this pack's background never exceeded 25 pages in an hour and read 500 every day; only the daily bucket caught it.

Algorithms

Algorithm	Behaviour	Memory per key	Boundary burst	Use it for
Fixed window	Counter resets at window edge	1 integer + window id	Up to 2× limit across the edge	Daily caps where 2× at midnight is acceptable
Sliding window log	Timestamp of every hit in the window	O(limit)	None	Exact small limits, low traffic
Sliding window counter	Weighted blend of current and previous fixed windows	2 integers	Near none	Hourly caps; what most libraries call "sliding window"
Token bucket	Bucket refills at rate r, holds up to b tokens; each hit takes one	2 numbers	Controlled: burst b then rate r	Human-friendly shaping (a visitor can open 10 tabs, then settle to a rate)
Leaky bucket / GCRA	Equivalent to token bucket expressed as a theoretical arrival time	1 timestamp	Controlled	Same as token bucket, single value to store

For page limits on a small site: sliding window counter for the hour, fixed window for the day (a UTC day, or the visitor's day if you key on cookie). Token bucket is better for APIs where the client can be told the refill rate.

Keys

• IP (IPv4): the default. Weak against a scraper with a proxy pool, strong against a single-machine headless crawler, and fair to most humans. Get the client IP from the right hop: on Cloud Run the client address is appended to X-Forwarded-For by the load balancer, so take the address that many hops from the right, never the leftmost value, which the client controls. Count your trusted proxies once and put the number in configuration.
• IPv6 /64: a single subscriber usually has a /64 (or /56); keying on the full address lets a client rotate through 2^64 addresses. Truncate.
• IPv4 /24: catches small pools and cloud tenants, and also catches a whole ISP's carrier-grade NAT block. Use only as a second, larger bucket (for example 10× the per-IP limit).
• ASN: aggregate by network owner (a MaxMind GeoLite2-ASN lookup, or ip.src.asnum at Cloudflare). A cap per hosting-provider ASN is a reasonable outer bucket because residential users are not on hosting ASNs. Requires a lookup on every request or at the edge.
• Session cookie: the crawler that uses a fresh profile per page never returns a cookie. Treat cookie-less clients as IP-keyed and cookie-bearing clients as cookie-keyed with a higher limit; this makes NAT-shared humans invisible to each other's counts while a stateless crawler stays IP-bound.
• Authenticated user: unlimited or very high for signed-in accounts you can suspend.

Sizing from Measurement

1. From request logs, count top-level document navigations per IP per hour and per day over 30 days, excluding known crawlers and your own monitoring.
2. Split by whether the IP had an engaged session (or two or more pages). Take percentiles of the engaged population: p99 and p99.9 per hour and per day.
3. Identify your shared-IP outliers: the IPs with the most distinct cookies or User-Agents. Their volume is many humans; your per-IP limit must sit above their daily peak or use cookie keying.
4. Profile the abuser the same way: pages per hour, pages per day, inter-request spacing.
5. Choose limits above the human p99.9 and below the abuser's daily volume. In the measured case the crawler did ~21 pages per hour and ~500 per day; limits of 60 per hour and 200 per day sat above the observed human distribution on that site (measure yours; do not copy these numbers) and below the crawler's daily volume. The hourly bucket never touched it. The daily bucket cut it to 200 pages a day, 40% of its former volume, with every further navigation that day answered 429; whether it then slows down, leaves, or spreads across addresses is what the counted-hit log tells you next.
6. Write the four numbers (human p99.9 hour and day, abuser hour and day) next to the limits in the code. The next person will need them.

Exempting Verified Crawlers

Search crawlers must not be limited, and a scraper must not be able to claim the exemption. Verify by IP, never by User-Agent alone:

• Googlebot: published ranges at developers.google.com/search/apis/ipranges/googlebot.json, or reverse DNS to googlebot.com / google.com followed by a forward lookup that returns the same IP.
• bingbot: published ranges at www.bing.com/toolbox/bingbot.json, or reverse DNS to search.msn.com with forward confirmation.
• Applebot: reverse DNS to applebot.apple.com with forward confirmation.
• OpenAI, and other vendors that publish JSON range lists: fetch the list on a schedule, cache it, match by CIDR.

Do the verification asynchronously or from a cached table; a DNS lookup inside a request path is a latency and cost problem. Unverified clients claiming a crawler token get the ordinary limit, not a block; they may be a misconfigured legitimate tool.

The 429 Response

• Status 429 Too Many Requests.
• Retry-After: <seconds> computed from the bucket reset, not a constant.
• Cache-Control: no-store.
• For Accept: text/html (and every navigation), an honest HTML body: what happened, the reset time, a contact address, and a sentence saying that declared crawlers can be exempted by request. A blank 429 teaches nothing and generates support email. For Accept: application/json, a small JSON body.
• Optionally the IETF draft RateLimit-Policy and RateLimit headers (draft-ietf-httpapi-ratelimit-headers); harmless and useful to good clients.
• Log every counted hit and every 429 with IP, key type, User-Agent, path and Sec-Fetch-Dest. You will need this to answer "did the limit hit a human".

Counters: In-Memory Versus Shared

An in-memory Map in the server process is free, fast and correct for exactly one instance. On a platform that scales horizontally:

• Each instance keeps its own count, so the effective limit is limit × instances if the load balancer spreads a client evenly, and somewhere between limit and limit × instances otherwise.
• Instances scale to zero and lose the count; a daily bucket evaporates at the first idle period.
• Cloud Run session affinity reduces the spread but is best-effort.

For a small site that usually runs one instance, in-memory counting catches the single-machine crawler because that crawler hits the same instance; document the caveat in the code. When you need correctness, use a shared store and price it per hit:

Store	Cost model	Notes
Redis (Upstash, Memorystore, self-hosted)	Upstash bills per command with a free daily allowance; a @upstash/ratelimit sliding window costs one or two commands per hit	The standard answer for serverless; sub-millisecond from the same region
Firestore	One read and one write per hit; daily free quota then per-100k pricing	Adds two round trips to every navigation and a sustained-write limit of about one write per second per document, which a burst breaks. Poor fit for hot keys
Edge KV (Cloudflare KV, Vercel Edge Config)	Eventually consistent	Fine for allow-lists and block-lists, wrong for counters
Durable Objects / Cloudflare Rate Limiting rules	Counted at the edge before the origin	Removes the origin from the loop entirely; see below

Whatever the store, keep the check off the asset path: match only HTML navigations so the twenty asset requests per page never touch the counter.

Framework Pitfalls

Next.js (App Router). Client-side navigations and prefetches are fetch() calls carrying RSC: 1, and prefetches carry Next-Router-Prefetch: 1. Next.js strips those internal headers before your middleware runs (in Next.js 16 the file is proxy.ts; the behaviour is the same), so middleware cannot distinguish a prefetch from a navigation by those headers. Key on the browser-owned fetch metadata instead: Sec-Fetch-Dest: document is a top-level navigation; empty is a fetch() (RSC payload, prefetch, beacon); image, script, style, font are assets. Browsers set these and page JavaScript cannot override them. Non-browser clients send no Sec-Fetch-* headers at all; count those as documents, because they are not prefetching anything.

// proxy.ts (Next.js 16) or middleware.ts — count only real page navigations
import { NextResponse, type NextRequest } from 'next/server';

export const config = { matcher: ['/skills/:path*'] };

export function proxy(req: NextRequest) {          // export `middleware` instead on Next.js 15 and earlier
  const dest = req.headers.get('sec-fetch-dest');
  if (dest && dest !== 'document') return NextResponse.next();       // prefetch, RSC, asset
  const ip = clientIp(req.headers.get('x-forwarded-for'), TRUSTED_HOPS);
  if (isVerifiedCrawler(ip)) return NextResponse.next();
  const ua = req.headers.get('user-agent') ?? '';
  const hour = bump(`h:${ip}`, 3600), day = bump(`d:${ip}`, 86400);
  log({ ip, ua, path: new URL(req.url).pathname, dest, hour, day });
  if (hour > 60 || day > 200) {
    return new NextResponse(html429, {
      status: 429,
      headers: { 'Retry-After': String(retryAfter(hour > 60 ? 'h' : 'd')), 'Cache-Control': 'no-store',
                 'Content-Type': 'text/html; charset=utf-8' },
    });
  }
  return NextResponse.next();
}

Also: the middleware matcher must exclude /_next/static, /_next/image, fonts and images, and any route behind ISR that a CDN serves without hitting you. Static prerendered pages still pass through middleware on Vercel and on a self-hosted Node server, so counting there is correct; on a pure CDN in front, counting has to move to the edge.

Edge and CDN products. Cloudflare Rate Limiting rules are available on every plan with plan-dependent limits on rule count, counting period and characteristics (Free is a single rule keyed on IP over a short period; higher plans add longer periods and keys such as headers, cookies and ASN; check the current plan table). Vercel Firewall custom rules support a rate-limit action on paid plans, keyed on IP or JA4 among others. Google Cloud Armor supports throttle and rate_based_ban rules, keyed on IP, forwarded IP, header, cookie, path or region, but only behind a Global external Application Load Balancer, which has its own monthly cost. Whichever you use, the edge rule and the origin rule should agree on what a "navigation" is; at the edge you typically match on path plus Accept containing text/html plus method GET.

Procedure: Testing Before Shipping

1. next build && next start (or your framework's production mode). Development servers add requests and disable prefetching; the numbers are wrong there.
2. Open the site in a real browser with the network panel open. Scroll a listing page so that 30 links prefetch. Confirm the counter did not move (read your log line).
3. Navigate through 10 pages by clicking. Confirm exactly 10 counted hits, all dest=document.
4. Hammer it from a shell and watch for the transition:

for i in $(seq 1 70); do
  curl -s -o /dev/null -w '%{http_code} ' -H 'Sec-Fetch-Dest: document' \
    -H 'User-Agent: test-limit/1.0' http://localhost:3000/skills/example-page
done; echo

5. Confirm the 61st response is 429, carries Retry-After, and has an HTML body.
6. Confirm the next request from a different source IP (or a different X-Forwarded-For at the correct hop in a trusted test setup) is not affected.
7. Confirm Googlebot-verified ranges bypass by feeding a known Googlebot IP through the same path in a test harness.
8. Deploy behind a feature flag or at 10× the intended limit for a day, read the counted-hit log, then tighten.

Checklist

• Only top-level document navigations are counted; prefetches, RSC fetches and assets are not.
• Client IP is taken from the correct X-Forwarded-For hop.
• Hourly and daily buckets, sized from the human p99.9 and the abuser's measured volume, with the numbers recorded in the code.
• Verified search crawlers exempt by IP; unverified claims get the normal limit.
• 429 has Retry-After, no-store, and an HTML body with contact details.
• Every counted hit is logged with IP, User-Agent, path and Sec-Fetch-Dest.
• Multi-instance behaviour documented; shared store priced per hit if used.
• Tested against a production build, in a browser and from a shell, before release.

Common Mistakes

• Counting every request. The human reading three pages trips the limit first.
• Trusting framework-internal headers that the framework strips before your code runs.
• Only an hourly limit, which a slow crawler never reaches.
• Reading the leftmost X-Forwarded-For value.
• Blocking on a User-Agent that says "Googlebot".
• Using Firestore as a hot-key counter and paying two operations for every page view.
• Shipping a 429 with no body and no Retry-After, then discovering it from a customer.
• Testing on the development server, where prefetch behaviour differs.

Limits

Rate limiting is proportional pressure, not identification. A crawler that spreads across a proxy pool slips under per-IP limits and needs ASN or fingerprint keys at the edge, or content gating. A limit tight enough to stop everything automated will stop humans behind shared addresses first. When the measured cost of the crawler is trivial and the harm is analytics contamination, the cheaper fix may be a delayed beacon and engaged-session reporting rather than a limit at all; this pack's cost and analytics skills make that call.