Bot Traffic and Crawler Defense skills for AI agents

Triggers when the user needs to find, profile or verify an automated client in access logs: grouping by IP, /24, ASN and User-Agent, reading Sec-Fetch headers, checking TLS fingerprints, confirming a claimed Googlebot, or reconstructing a crawler's sessions. Trigger on "access logs," "who is this IP," "crawler in the logs," "verify Googlebot," "reverse DNS," "JA3," "JA4," "Sec-Fetch-Dest," "Logs Explorer query," "BigQuery logs," "CloudFront logs," "scraper IP," or "bot traffic analysis." Covers Cloud Logging, Vercel, nginx, Caddy and CloudFront formats, jq, awk and SQL snippets, RDAP and whois for ownership, and the sampling and cost of logging itself.

Triggers when the user suspects their analytics are inflated by automation, or asks why GA4, Plausible, Fathom or their own server-side counters show users, pageviews or a "most viewed" list that does not match reality. Trigger on "bot traffic," "GA4 users vs engaged sessions," "direct traffic spike," "crawler in analytics," "headless browser," "fake pageviews," "engagement rate," "IAB bot filter," "Plausible bots," or "most viewed is wrong." Covers the one-page/under-ten-seconds/new-user-every-hit signature, hourly steadiness, single OS-browser-device fingerprints, uniform per-page counts, GA4 comparisons, data filters and Data API queries, why IAB filtering misses JavaScript-executing headless browsers, and what to report instead.

Activate this skill when the user is writing a crawler, scraper or fetcher of any size and wants it to be identifiable, cheap for the sites it visits, resilient to rate limiting, and defensible. Triggers on "build a crawler," "web scraping etiquette," "polite scraper," "obey robots.txt," "Crawl-delay," "conditional requests," "ETag If-Modified-Since," "back off on 429," "headless browser scraping," "scraper legal," "CFAA hiQ," "GDPR scraping," or "terms of service scraping." Covers self-identification, robots.txt parsing, rate and concurrency limits with jitter, caching and revisit policy, sitemaps before spidering, Retry-After handling, preferring HTML or APIs over headless browsers, and an outline of the legal landscape with an explicit not-legal-advice line naming the professional to consult.

Triggers when the user wants to know exactly what their site hands to any client that asks, or wants to gate full text behind a preview without destroying search visibility. Trigger on "content exposure," "web scraping protection," "preview gating," "show 30%," "paywall structured data," "RSC payload," "__NEXT_DATA__," "sitemap leaks," "public JSON," "canary text," "watermark content," "scraper copied my site," or "API keys and quotas." Covers every path full text ships on (server-rendered HTML, React Server Component payloads, prerendered routes, JSON data files, sitemaps, public APIs, RSS), why client-side gates are theatre, server-side preview gating with indexable summaries, API-first access, canary sentences, and the SEO and AI-search cost of hiding text.

Activate this skill when the user wants to know what automated traffic actually costs them on pay-per-use hosting, or is deciding whether a crawler is a billing problem, a data problem, or no problem at all. Triggers on "cloud bill," "bot traffic," "crawler cost," "Cloud Run pricing," "Vercel usage," "Lambda invocations," "egress," "free tier," "web scraping cost," or "how much is this scraper costing me." Covers per-request, CPU-time, memory, egress, image optimisation and database charges triggered per page, where free tiers end, a fully worked arithmetic example, and how to measure from billing exports and request logs.

Activate this skill when the user is choosing or configuring an edge product to handle bot traffic: Cloudflare Bot Fight Mode, Super Bot Fight Mode, Bot Management, Turnstile or managed challenges; Google Cloud Armor; AWS WAF Bot Control; Vercel Firewall and Attack Challenge Mode; or Fastly. Triggers on "WAF," "bot management," "Cloudflare challenge," "Turnstile," "Cloud Armor pricing," "AWS WAF Bot Control," "Vercel Firewall," "Attack Challenge Mode," "block scraper at the edge," or "is this worth $5 a month." Covers what each product actually does and charges for, what JavaScript challenges do to search crawlers, AI crawlers, link previews and accessibility, when a control costs more than the crawler, and a rule order that allow-lists verified crawlers first, challenges by score, and blocks only on evidence.

Activate this skill when the user is writing or auditing a robots.txt, deciding which search, AI-training and AI-assistant crawlers to allow, or asking whether a crawler will obey it. Triggers on "robots.txt," "crawler," "User-agent," "Disallow," "Crawl-delay," "Googlebot," "GPTBot," "ClaudeBot," "Google-Extended," "CCBot," "Bytespider," "llms.txt," "AI crawler policy," or "block AI bots." Covers RFC 9309 syntax and precedence, wildcard and Sitemap support by engine, the crawler tokens that matter in 2026 grouped by purpose, the visibility trade-off of each group, testing with curl and Search Console, and what robots.txt cannot do against an undeclared headless browser or scraper.

Activate this skill when the user must decide which crawlers to admit and which to refuse, and wants the decision grounded in what each class returns: search traffic, AI citations, nothing, or copies and a cloud bill. Triggers on "should I block GPTBot," "AI Overviews traffic," "crawl budget," "Search Console crawl stats," "AI search referrals," "ChatGPT citations," "Perplexity referrals," "Google-Extended," "crawler policy," "bot traffic policy," or "robots.txt strategy." Covers the four crawler classes and their return, what blocking does to indexing and to AI answer citations, measuring referrals from AI surfaces, structured data and canonical signals, a policy matrix small sites can adopt, and a quarterly review as tokens change.

Activate this skill when the user needs to design or debug rate limiting for page traffic on a public site, as opposed to an API: sizing limits from real user percentiles, choosing keys and windows, deciding between in-memory and shared counters, exempting search crawlers, and returning a correct 429. Triggers on "rate limiting," "rate limit middleware," "token bucket," "sliding window," "429 Retry-After," "per-IP limit," "Next.js middleware rate limit," "Sec-Fetch-Dest," "Upstash ratelimit," "Cloudflare rate limiting rule," or "bot traffic keeps reading every page." Covers hourly plus daily buckets, multi-instance effects on the effective limit, framework pitfalls, and testing against a production build before shipping.