You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have turned on every edge bot product at least once, read the bill and the Search Console coverage report afterwards, and turned some of them off again. You know what a managed challenge does to a link preview, what a load balancer costs before the WAF on top of it charges anything, and you have watched a $2-a-month crawler justify a $200-a-month stack. You size the control to the measured problem.

## Key Points

- **Turnstile**: a free CAPTCHA replacement widget you embed on your own pages and verify server-side with the `siteverify` endpoint. Protects a form or an action, not a route.
2. **Allow your own paths and clients.** Uptime monitor IPs, webhook senders, `/api/` routes used by your SDK, `/robots.txt`, `/sitemap.xml`, `/.well-known/`.
4. **Challenge by score**, only if the product gives you a score. Managed challenge for low scores on HTML paths, never on assets, never on API paths.
5. **Block on evidence.** A specific IP or /24 you have profiled in the logs and verified as hostile, with an expiry date on the rule.
6. **Log everything else.** Use a log or count action as the last rule for a week before tightening any of the above.
- Search Console crawl stats and page indexing: no rise in `403`, `429` or `5xx` responses to Googlebot; no drop in crawled pages per day.
- Paste a page URL into a chat tool and a social composer and confirm the preview card still renders.
- The uptime monitor has not paged.
- Your own request logs: the crawler's daily page count, before and after.
1. Cost the crawler (the cost-accounting skill). Write the monthly number down.
2. List the non-monetary harms: analytics contamination, ranked-list poisoning, content copies.
4. Write the visibility risk: which rows in the table above you would affect and how you would exempt them.

Edge Bot Management and WAF

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have turned on every edge bot product at least once, read the bill and the Search Console coverage report afterwards, and turned some of them off again. You know what a managed challenge does to a link preview, what a load balancer costs before the WAF on top of it charges anything, and you have watched a $2-a-month crawler justify a $200-a-month stack. You size the control to the measured problem.

Core Principle: The Product Must Cost Less Than the Problem, Including Visibility

Edge bot products are sold on fear and priced on plans. For a small site paying per request, three numbers decide whether one belongs in front of your origin: the measured monthly cost of the automated traffic, the monthly price of the control (including any load balancer or plan upgrade it requires), and the value of the visibility the control can accidentally destroy. A challenge page that keeps out a $1.40 crawler and also keeps out Slackbot-LinkExpanding, ChatGPT-User and a screen-reader user who cannot solve the interactive fallback has a negative return.

The corollary: for most small sites the edge product's first value is not blocking. It is the verified-bot list, the fingerprint fields and the logs, which let you write precise rules cheaply. Blocking comes last and only on evidence.

What Each Product Actually Does

Cloudflare

• Bot Fight Mode (Free plan): challenges requests that Cloudflare classifies as definitely automated, zone-wide, with no path exclusions and no way to write a skip rule around it. It is known to challenge legitimate API clients and webhook senders. Verified bots are allowed. Free, blunt, and the reason people discover their uptime monitor is being challenged.
• Super Bot Fight Mode (Pro and Business): per-category actions (definitely automated, and on Business also likely automated: allow, block, managed challenge), a verified-bots toggle, static-resource protection, and optional JavaScript detections. Still zone-wide with limited exclusions.
• Bot Management (Enterprise): a per-request bot score (1 to 99), verified-bot flag and category, JA3/JA4 hashes and detection IDs exposed as fields (cf.bot_management.score, cf.bot_management.verified_bot, cf.bot_management.ja4) usable in custom rules. Priced by contract. This is the product that lets you "challenge by score"; the lower tiers do not expose a score.
• WAF custom rules (all plans, count varies by plan): match on IP, ASN (ip.src.asnum), country, path, headers, cf.client.bot (verified bot, available on all plans), and more; actions include block, managed challenge, JS challenge, interactive challenge, log, skip. Free plan includes a small number of custom rules and one rate-limiting rule; check the current plan table.
• Managed Challenge: picks a non-interactive check when the client looks like a browser and escalates to interactive when it does not. Passing sets a cf_clearance cookie for the configured challenge passage period. A fresh-profile-per-page crawler never carries the cookie and is challenged every time, which is the point.
• Turnstile: a free CAPTCHA replacement widget you embed on your own pages and verify server-side with the siteverify endpoint. Protects a form or an action, not a route.
• AI crawler controls: a per-zone toggle to block known AI crawlers on all plans (introduced 2024), an AI Crawl Control dashboard with per-crawler allow and block (2025), a default-block for AI crawlers on new zones (2025), and an experimental "pay per crawl" scheme. Check the current dashboard; these move quickly. Note that "known AI crawlers" means declared tokens; an undeclared headless browser is not on the list.
• AI Labyrinth (2025): serves generated decoy pages to disallowed crawlers instead of blocking them. Costs you nothing per request; whether you want to feed a crawler nonsense is a policy question.

Google Cloud Armor

Security policies attach to backend services behind a Global external Application Load Balancer. A Cloud Run service reaches that load balancer through a serverless network endpoint group, which means the prerequisite for using Cloud Armor on a small Cloud Run site is a load balancer with its own fixed monthly forwarding-rule charge (on the order of $18 a month; check the current figure) before any Armor pricing. Cloud Armor Standard bills per security policy per month, per rule per month, and per million requests evaluated (historically $5, $1 and $0.75; check the current Cloud Armor pricing page). Rules support IP, CIDR, country, custom expressions on headers and paths, preconfigured OWASP rule sets, rate-based throttle and rate_based_ban actions keyed on IP, forwarded IP, header, cookie, path, SNI or region, and integration with reCAPTCHA Enterprise for challenges, where assessments beyond the free monthly allowance are billed separately. Adaptive Protection and the Enterprise tier are subscription products for larger fleets. For a site under the free tier, the load balancer alone is ten times the crawler.

AWS WAF and Bot Control

A web ACL attaches to CloudFront, an Application Load Balancer, API Gateway or App Runner. Pricing is per web ACL per month, per rule per month and per million requests (historically $5, $1 and $0.60; check the current figure). The Bot Control managed rule group adds a monthly fee plus a per-million-request charge, with two levels: Common (signals such as header consistency and a verified-bot allow list, labels like awswaf:managed:aws:bot-control:bot:category:search_engine) and Targeted (adds machine-learning detection, browser challenge and CAPTCHA actions, and dynamic rate scoping, at a higher per-request price). Challenge and CAPTCHA actions bill per attempt analysed. Labels let you write a rule that counts rather than blocks, which is the right first step.

Vercel Firewall

Included on every plan: DDoS mitigation, Attack Challenge Mode (a toggle that challenges every visitor, including every crawler, so it is an emergency switch, not a setting), and custom rules with plan-dependent counts. Rules match on path, method, IP, ASN, country, User-Agent, headers, cookies, query, JA3, JA4 and more; actions are deny, challenge, log, bypass and, on paid plans, rate limit. Managed rule sets include bot protection and an AI-bots rule set for declared crawlers; check the current dashboard for what is included at which plan and whether rate limiting is metered. Persistent actions can block an IP for a period after a rule fires.

Fastly

Next-Gen WAF (from the Signal Sciences acquisition) and Fastly Bot Management are enterprise-priced contracts; Fastly's edge compute lets you write your own logic, which is only cheaper if you already run on Fastly. Not a first choice for a site that fits in a free tier.

What Challenges Do to the Traffic You Want

Client	Effect of a JS or managed challenge	Effect of a block by score
Googlebot, bingbot, Applebot	Cannot pass; pages report 403/503 in Search Console and drop from the index over time. Only safe if verified bots are exempted first	Same, unless exempted
OAI-SearchBot, Claude-SearchBot, PerplexityBot	Cannot pass; your pages leave those indexes	Same
ChatGPT-User, Claude-User, Perplexity-User	Cannot pass; the assistant tells the user the page could not be read; you lose the citation	Same
Link unfurlers (Slackbot-LinkExpanding, Twitterbot, facebookexternalhit, Discordbot, TelegramBot, WhatsApp, LinkedInBot)	No preview card; shares show a bare URL	Same
Uptime monitors and webhook senders	Paged for false downtime; failed deliveries	Same
RSS readers, curl users of your docs, package managers	Broken	Same
Screen-reader and keyboard users	Non-interactive challenges usually pass; interactive fallbacks are a barrier	Not affected
VPN, Tor, Linux and privacy-browser users	Repeated challenges; some leave	Blocked or challenged repeatedly by low score
The headless-Chrome catalogue crawler	Fails the challenge or is slowed to nothing by re-challenge on every fresh profile	Blocked if its score is low; a real Chrome engine often scores mid-range

The vendors' verified-bot lists cover most rows in the top half, but only if your rule order skips verified bots before the challenge rule, and only for bots the vendor has verified; new AI fetchers lag the list.

Rule Design

Order matters; edge rule engines evaluate top to bottom and the first terminal action wins.

1. Allow verified crawlers. cf.client.bot on Cloudflare; the Bot Control verified_bot label on AWS; the Vercel bot-protection managed rule's verified list. Search first, then AI-search and assistant fetchers you have decided to admit.
2. Allow your own paths and clients. Uptime monitor IPs, webhook senders, /api/ routes used by your SDK, /robots.txt, /sitemap.xml, /.well-known/.
3. Rate limit HTML navigations per IP (and per JA4 or ASN if the product offers the key), sized from the measured human percentiles as in the rate-limiting skill. Match GET, path prefix, and Accept containing text/html.
4. Challenge by score, only if the product gives you a score. Managed challenge for low scores on HTML paths, never on assets, never on API paths.
5. Block on evidence. A specific IP or /24 you have profiled in the logs and verified as hostile, with an expiry date on the rule.
6. Log everything else. Use a log or count action as the last rule for a week before tightening any of the above.

A worked rule set for the measured crawler (one address, one country, fresh profile each page, stock Chrome) needed no product beyond origin rate limiting: 60 navigations an hour and 200 a day per IP, verified crawlers exempt. Had it been a proxy pool, step 3 keyed on JA4 at the edge and step 5 on the hosting ASN would have been the next two moves, in that order.

Worked Example: Pricing the Control Against the Crawler

The measured crawler: ~315,000 requests a month, ~5 GB egress, under $2 on the invoice. The monthly cost of each way to stop it, using the mechanisms above with figures as of the measurement period (check every one on the vendor's pricing page):

Control	Fixed per month	Variable at 315k requests	Approximate total	Ratio to the crawler's cost
Origin middleware rate limit (hourly and daily per IP)	$0	$0	one or two engineer hours, once	small but not zero
Cloudflare Free: custom rules, one rate-limiting rule, AI-crawler toggle	$0	$0	DNS migration effort	small
Cloudflare Pro with Super Bot Fight Mode	on the order of $20 to $25	$0	$20 to $25	10 to 15×
Google Cloud Armor Standard behind a Global external Application Load Balancer	forwarding rule ~$18, policy ~$5, ~$1 per rule	~$0.75 per million requests, so ~$0.25	~$25 to $30	15×
AWS WAF with Bot Control Common on CloudFront	web ACL ~$5, ~$1 per rule, Bot Control ~$10	~$0.60 plus ~$1 per million, so ~$0.50	~$17 to $20	10×
Vercel Pro custom rules with rate limiting	Pro plan per seat, ~$20	plan-dependent metering	~$20 plus	10×

The origin rate limit won because the crawler was one address with no cookies. The Free-plan edge options are the next step if it becomes a pool; the paid rows are justified only if the site wanted the plan for other reasons or the crawler's cost grows by an order of magnitude.

Example Rule Set (Cloudflare Expression Syntax)

# 1. Skip everything below for verified bots
#    action: skip (remaining custom rules and rate limiting rules)
(cf.client.bot)

# 2. Skip for monitoring and machine paths
(ip.src in {198.51.100.10 198.51.100.11})
  or (http.request.uri.path in {"/robots.txt" "/sitemap.xml"})
  or starts_with(http.request.uri.path, "/api/")
  or starts_with(http.request.uri.path, "/.well-known/")

# 3. Rate limiting rule: HTML navigations on content paths, keyed on IP
(http.request.method eq "GET"
  and starts_with(http.request.uri.path, "/skills/")
  and any(http.request.headers["accept"][*] contains "text/html"))
#    characteristics: IP; period and threshold from the human p99.9 (the plan caps the period)
#    action: block, with a mitigation timeout

# 4. Managed challenge for hosting networks on content paths, only after a week of log-only
(starts_with(http.request.uri.path, "/skills/") and ip.src.asnum in {64496 64497} and not cf.client.bot)
#    action: managed_challenge

# 5. Block on evidence, with the reason and expiry in the rule description
(ip.src in {203.0.113.0/24})
#    action: block   description: "catalogue crawler; log query 2026-09-01; remove 2026-12-01"

# 6. Log-only catch-all on plans that expose a score
(cf.bot_management.score lt 30 and starts_with(http.request.uri.path, "/skills/"))
#    action: log

The same structure on AWS WAF: the Bot Control rule group first with its verified-bot labels, the whole group in count mode for the first week, then a rate-based rule scoped to /skills/ and a label-match rule that challenges only on HTML paths. On Vercel: rules 2, 3 and 5 in that order, with bypass for the monitor addresses, rate limit on the content path, and deny with an expiry note in the rule name.

Reading the Result After a Week

• The edge product's event log (Cloudflare Security Events, AWS WAF sampled requests, Vercel Firewall activity): which rule fired, on what, how often. A rule that never fires is a rule to delete; one that fires on a verified bot is an ordering bug.
• Search Console crawl stats and page indexing: no rise in 403, 429 or 5xx responses to Googlebot; no drop in crawled pages per day.
• Paste a page URL into a chat tool and a social composer and confirm the preview card still renders.
• The uptime monitor has not paged.
• Your own request logs: the crawler's daily page count, before and after.

Procedure: Deciding Whether to Buy

1. Cost the crawler (the cost-accounting skill). Write the monthly number down.
2. List the non-monetary harms: analytics contamination, ranked-list poisoning, content copies.
3. Price the control fully: plan upgrade, load balancer, per-request fees at your volume, rule and policy fees, challenge or assessment fees, and the hours to configure and to monitor Search Console afterwards.
4. Write the visibility risk: which rows in the table above you would affect and how you would exempt them.
5. If the origin can do it (rate limiting in middleware, delayed beacons, verified-crawler exemption), do that first and measure for two weeks.
6. Buy the edge product when the origin cannot see what it needs (TLS fingerprints, ASN at scale, DDoS volumes) or when the crawler's cost exceeds the control's cost plus an hour of your time a month.
7. Deploy in log-only mode for a week, read the matches, then switch one rule at a time to challenge or block.
8. Check Search Console crawl stats and coverage, and your link previews, one week after each change.

Checklist

• Crawler cost and control cost, both monthly, written side by side.
• Verified-crawler allow rule is above every challenge or block rule.
• Assets, API, robots.txt, sitemap and well-known paths excluded from challenges.
• Uptime monitors and webhook senders exempted by IP.
• Every block rule cites the log evidence and carries an expiry.
• Attack Challenge Mode and Bot Fight Mode understood as zone-wide, not tuned.
• Search Console and link previews rechecked after the change.

Common Mistakes

• Enabling a zone-wide challenge to stop one crawler and losing every link preview.
• Assuming "bot score" exists on a plan that does not expose it.
• Putting a Cloud Run service behind a load balancer for Cloud Armor without noticing the load balancer's fixed monthly price.
• Challenging API paths.
• Blocking a cloud ASN outright and taking out users on that provider's VPN.
• Buying the Enterprise tier for a site inside every free tier.
• Never revisiting rules; the block written for a 2024 crawler still fires on a 2026 customer.

Limits

Edge products see headers, fingerprints and volume; they do not know whether a real browser engine is being driven by a person or a script, and a well-built headless crawler scores like a browser because it is one. They also sit in front of your origin, so they cannot fix what the origin ships: if a JSON file hands over the whole catalogue in one request, no WAF rule sized for page traffic notices. Content exposure, rate limiting at the origin and the visibility policy are separate decisions covered elsewhere in this pack; this skill is only about whether an edge control is worth its price and how to order its rules if it is.