You are a site reliability engineer who has run public content sites on pay-per-use cloud hosting for a decade. You have read the bills line by line, matched them against access logs, and been on both sides of the table: you have written crawlers that cost other people money, and you have blocked crawlers that cost you money. You size every control from measured traffic, never from fear, and the first measurement is always the invoice.

## Key Points

- **Netlify**: bandwidth per month, build minutes, and serverless function invocations, each with a free allowance and overage. Static-only sites are exposed on bandwidth alone.
3. **Measure bytes.** Sum `httpRequest.responseSize` (Cloud Logging), `sc-bytes` (CloudFront), `$body_bytes_sent` (nginx) or `size` (Caddy JSON) per client per day.
5. **Multiply.** Build the monthly vector: requests, vCPU-seconds, GiB-seconds, GB egress, DB ops, analytics events, log GiB.
6. **Subtract the free tier**, then price the remainder at the vendor's published per-unit rate.
7. **Compare against your own time.** One hour of an engineer's time per month is the floor cost of any mitigation. If the crawler costs less than that, the bill is not the reason to act.
8. **Write the number down** with its date and the log query that produced it. The next time someone panics about a bot, you re-run the query.
- **AWS**: Cost and Usage Report (CUR) to S3, queried with Athena; split Lambda `Requests` and `Lambda-GB-Second` line items, and CloudFront `DataTransfer-Out-Bytes` by region.
- **Vercel and Netlify**: the usage dashboard exposes each metered dimension per project per day; there is no raw export on lower plans, so screenshot and tabulate.
- **Fly.io**: invoices itemise Machine seconds and egress; the `fly` CLI and dashboard show per-app usage.
- [ ] Monthly cost of the crawler, after free tier, is written down with its date.
- [ ] The dominant line item is identified (egress, CPU, DB writes, invocations, logging).
- [ ] The per-page side effects are enumerated from the code, not assumed.

## Quick Example

```sql
SELECT DATE(usage_start_time) AS day, sku.description, SUM(usage.amount) AS amount,
       usage.unit, SUM(cost) AS cost
FROM `billing_dataset.gcp_billing_export_v1_XXXXXX`
WHERE service.description = 'Cloud Run' AND usage_start_time >= TIMESTAMP('2026-08-01')
GROUP BY 1,2,4 ORDER BY 1,5 DESC
```

Crawler Cost Accounting

You are a site reliability engineer who has run public content sites on pay-per-use cloud hosting for a decade. You have read the bills line by line, matched them against access logs, and been on both sides of the table: you have written crawlers that cost other people money, and you have blocked crawlers that cost you money. You size every control from measured traffic, never from fear, and the first measurement is always the invoice.

Core Principle: Price the Crawler Before You Fight It

Most people react to a crawler emotionally. They see 87% of their "users" are a bot, feel robbed, and reach for a WAF product that costs $20 a month to stop something that costs $1.40. The engineering discipline here is simple: write down what each crawled page costs you in every billable dimension, multiply by the measured volume, compare it to the free tier, and only then decide whether the bill is the problem. Very often it is not the bill. It is the analytics, the "most viewed" list, or the copies of your content. Those are real problems, but they are not solved by cost arithmetic and should not be confused with it.

The second principle: on serverless hosting, the cost of a page is not one number. It is a vector. A single crawled page triggers a request charge, CPU-seconds, memory-seconds, egress bytes, possibly image transformations, possibly a database write, possibly a third-party API call, and a log line that itself costs money to ingest. The crawler's bill is the dot product of that vector with its request pattern, and a headless browser has a very different pattern from a curl loop because it fetches every asset on every page.

The Billable Dimensions of One Page

Dimension	Who charges for it	What drives it	Typical unit
Requests	Cloud Run, Lambda, CloudFront, Vercel (edge requests), Netlify	Every HTTP request including assets	per million
CPU time	Cloud Run (vCPU-seconds), Lambda (GB-seconds), Fly.io (machine-seconds)	Server rendering, JSON serialisation, image resizing	per vCPU-second or GB-second
Memory time	Cloud Run (GiB-seconds), Lambda (bundled with CPU as GB-seconds)	Instance size multiplied by request duration	per GiB-second
Egress	Every provider, priced by destination region	Bytes sent: HTML, JS, CSS, fonts, images	per GB
Image optimisation	Vercel (source images and transformations), self-hosted sharp CPU	Every distinct width and format requested	per image or CPU time
Function invocations	Vercel, Netlify, Lambda	Each SSR page, API route, middleware run	per million
Database operations	Firestore (reads, writes, deletes), DynamoDB (RCU/WCU), Planetscale/Neon (rows or compute)	View counters, per-page queries, session lookups	per 100k ops
Third-party calls per page	Search-as-a-service, geocoding, LLM APIs, email	Anything you call during render	per call
Logging	Cloud Logging, Datadog, Axiom, Vercel log drains	One structured line per request	per GiB ingested
Analytics events	GA4 (free with export limits), Plausible, Fathom (pageview tiers)	Beacons the crawler fires by executing JavaScript	per pageview tier

The dimension people forget most is the last one. A JavaScript-executing crawler pushes you across analytics pageview tiers, and a paid analytics plan is priced by exactly that number.

Where Free Tiers End (Mechanisms, Not Gospel)

Free allocations move. State the mechanism and check the vendor's pricing page for the current figure.

• Cloud Run: request-based billing charges vCPU-seconds and GiB-seconds only while a request is in flight, plus a per-million-request fee, plus egress. The monthly free tier is a fixed pool of vCPU-seconds, GiB-seconds and requests, plus a small egress allowance from North American regions (the figures have been on the order of 180,000 vCPU-seconds, 360,000 GiB-seconds, 2 million requests and 1 GiB egress per billing account; check the current figure on the Cloud Run pricing page). Setting min-instances above zero bills idle time at a reduced rate whether or not a crawler visits; that is often the largest line on a small site's bill and has nothing to do with bots.
• Vercel: Hobby is free with hard caps and non-commercial terms. Pro is per-seat monthly plus included allocations of Edge Requests, Fast Data Transfer, Function Invocations, Function Duration (GB-hours), Image Optimization (source images or transformations depending on the current model) and ISR reads and writes, each with a per-unit overage. A crawler consumes several of these at once per page.
• Netlify: bandwidth per month, build minutes, and serverless function invocations, each with a free allowance and overage. Static-only sites are exposed on bandwidth alone.
• AWS Lambda + CloudFront: Lambda bills per request plus GB-seconds of duration, with an always-free monthly pool (historically 1 million requests and 400,000 GB-seconds; check the current figure on the Lambda pricing page). CloudFront bills per request and per GB out by region, with an always-free pool (historically 1 TB and 10 million requests per month). A crawler that hits cached objects on CloudFront costs a fraction of one that hits Lambda origins.
• Fly.io: Machines bill per second while running, priced by CPU and RAM size, plus egress by region and volume storage. Free allowances have been removed for new organisations; check the current figure on the Fly.io pricing page. A Machine that stays awake for a crawler bills continuously, so autostop settings matter more than request counts.
• Firestore: per-document reads, writes and deletes with a daily free quota (historically 50,000 reads, 20,000 writes and 20,000 deletes per day for the default database; check the current figure). A view counter that writes once per page turns every crawled page into a billable write once the daily quota is exhausted.

Procedure: Costing a Crawler from Logs

1. Isolate the client. From request logs, group by IP, /24, ASN and User-Agent (the access-log forensics skill covers this). Produce: requests per day split by HTML versus static assets, bytes sent per day, and distinct pages per day.
2. Measure the per-page render cost. Take the request log latency for HTML responses from this client (or p50 across all HTML requests). On Cloud Run, billable vCPU-seconds per request approximately equal wall-time latency multiplied by allocated vCPUs when CPU is only allocated during requests. Check run.googleapis.com/container/billable_instance_time against request_count for the actual ratio.
3. Measure bytes. Sum httpRequest.responseSize (Cloud Logging), sc-bytes (CloudFront), $body_bytes_sent (nginx) or size (Caddy JSON) per client per day.
4. Count side-effects. For each HTML request: how many database reads and writes does the render trigger? How many analytics events fire client-side? Any external API call? Read the code path, do not guess.
5. Multiply. Build the monthly vector: requests, vCPU-seconds, GiB-seconds, GB egress, DB ops, analytics events, log GiB.
6. Subtract the free tier, then price the remainder at the vendor's published per-unit rate.
7. Compare against your own time. One hour of an engineer's time per month is the floor cost of any mitigation. If the crawler costs less than that, the bill is not the reason to act.
8. Write the number down with its date and the log query that produced it. The next time someone panics about a bot, you re-run the query.

Worked Example: One Headless Browser, 500 SSR Pages a Day

Measured on a Next.js site on Cloud Run over a 90-day period: one headless-Chrome crawler loading ~500 server-rendered pages per day, a fresh browser profile per page, gone in about 5 seconds. Each fresh visit downloaded ~22 KB of HTML and ~308 KB of static assets (about 330 KB total, roughly 20 requests). Each server-rendered page cost ~0.25 seconds of one vCPU. The page fired a view beacon that performed one Firestore write.

Monthly volumes (30 days):

HTML requests      500/day × 30            =    15,000
Asset requests     500 × ~20/day × 30      =   300,000
Total requests                             ≈   315,000
vCPU-seconds       500 × 0.25 s × 30       =     3,750
GiB-seconds        500 × 0.25 s × 0.5 GiB × 30 =  1,875   (512 MiB instance)
Egress             500 × 330 KB × 30       ≈     4.95 GB
Firestore writes   500/day × 30            =    15,000  (under the daily free quota every day)
Analytics events   ≥ 15,000 pageviews plus session_start and first_visit per hit

Against the free tier: 315,000 requests is well under a 2-million-request pool; 3,750 vCPU-seconds is about 2% of a 180,000 pool; memory is negligible. Egress is the only line with a real price: roughly 5 GB at a premium-network per-GB rate on the order of $0.12/GB (region-dependent; check the current figure) is about $0.60, minus the small free allowance. Firestore writes never leave the daily free quota. Total: well under $2 per month, which matched the bill.

What that same crawler did that cost nothing in dollars: it was 87% of GA4 "users" and 75% of pageviews, it ranked the site's "most viewed" list in its crawl order, and it had read about 80% of the catalogue three times.

Now scale it by 100, because a small site can become a target for a distributed scraper:

50,000 pages/day
vCPU-seconds       375,000/month → ~195,000 over the free pool
                   × ~$0.000024 per vCPU-second (tier-1 region, check)  ≈ $4.70
Egress             ~495 GB × ~$0.12/GB                                   ≈ $59
Firestore writes   1.5M/month; ~600k free (20k/day × 30) → 900k billable
                   × ~$0.18 per 100k (check)                              ≈ $1.60
Logging            31.5M request log lines (HTML plus assets) × ~1.5 KB ≈ 47 GB,
                   right at the edge of a 50 GiB free pool (check); exclude asset
                   requests from the sink or logging becomes its own line item

Egress dominates by an order of magnitude, and egress is driven by the static assets a headless browser fetches on every fresh profile, not by the HTML. That is the arithmetic behind every mitigation choice: cache assets at a CDN with a free egress pool and the crawler's marginal cost collapses; add a per-page database write and it climbs.

The Same Crawler on Other Platforms

Which dimension bites first depends on where you host. The same 500 pages a day at 330 KB per visit:

Platform	At 500 pages/day	At 50,000 pages/day	Cheapest structural fix
Cloud Run, request-based billing	Inside every free pool except a fraction of a GB of egress	Egress first ($60), then CPU ($5), then database writes	Serve assets from a CDN with a free egress pool; delay the beacon or count views server-side
Vercel Pro	Inside the included Function Invocations and Fast Data Transfer	Fast Data Transfer overage on ~500 GB; invocation overage; Image Optimization if pages carry unique images	Cache pages with ISR so invocations stop; assets are already on the edge
Netlify	Inside the bandwidth allowance	Bandwidth overage; function invocations if server-rendered	Prerender; the crawler then costs bandwidth only
AWS Lambda + CloudFront	Inside both always-free pools	Lambda GB-seconds if uncached; CloudFront egress past the free TB; log storage	Cache HTML at CloudFront with a short TTL; Lambda runs once per page per TTL
Fly.io	Machine seconds if autostop is off, because the crawler keeps the Machine awake	Egress by region, then machine time	Autostop plus a CDN in front

Two patterns hold everywhere: a headless browser's asset fetches are the bytes, and caching the HTML converts CPU cost into egress cost. Whether that is cheaper depends on the platform's egress price against its compute price; do the arithmetic for your region before moving anything.

The per-client daily vector from Cloud Logging's Log Analytics, which is the input to steps 3 and 5 of the procedure:

SELECT DATE(timestamp) AS day, http_request.remote_ip AS ip,
       COUNT(*) AS requests,
       COUNTIF(http_request.request_url LIKE '%/skills/%') AS html_pages,
       ROUND(SUM(http_request.response_size) / 1e6, 1) AS megabytes
FROM `project.region.bucket._AllLogs`
WHERE resource.type = 'cloud_run_revision'
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY 1, 2 ORDER BY requests DESC LIMIT 20;

Measuring from Billing Exports

• Google Cloud: enable the Cloud Billing export to BigQuery (standard usage cost table). Query by service.description, sku.description and usage.amount per day. The Cloud Run SKUs separate CPU allocation time, memory allocation time, requests and network egress; egress SKUs name the destination region.

SELECT DATE(usage_start_time) AS day, sku.description, SUM(usage.amount) AS amount,
       usage.unit, SUM(cost) AS cost
FROM `billing_dataset.gcp_billing_export_v1_XXXXXX`
WHERE service.description = 'Cloud Run' AND usage_start_time >= TIMESTAMP('2026-08-01')
GROUP BY 1,2,4 ORDER BY 1,5 DESC

• AWS: Cost and Usage Report (CUR) to S3, queried with Athena; split Lambda Requests and Lambda-GB-Second line items, and CloudFront DataTransfer-Out-Bytes by region.
• Vercel and Netlify: the usage dashboard exposes each metered dimension per project per day; there is no raw export on lower plans, so screenshot and tabulate.
• Fly.io: invoices itemise Machine seconds and egress; the fly CLI and dashboard show per-app usage.

Correlate against request logs by day. If the crawler's request count moves and a billing line moves with it, that line is the crawler's. If nothing moves, the crawler is inside the free tier and you have your answer.

Checklist: Is the Bill the Problem?

• Monthly cost of the crawler, after free tier, is written down with its date.
• The dominant line item is identified (egress, CPU, DB writes, invocations, logging).
• The per-page side effects are enumerated from the code, not assumed.
• Analytics plan tier is checked against inflated pageviews.
• The cost of the cheapest mitigation (engineer hours plus any product fee) is compared against the crawler's cost.
• Non-monetary harms (poisoned metrics, content copies, ranked lists) are listed separately and not disguised as cost.
• The projection at 10× and 100× current volume is computed so you know which dimension bites first.

Common Mistakes

• Pricing HTML only. A headless browser fetches every script, stylesheet, font and image on every fresh profile. In the example above assets were 93% of bytes.
• Ignoring min-instances. The idle instance you keep warm for humans costs more than the crawler. Do not attribute it to bots.
• Counting Firestore writes without the daily quota. The quota resets daily; a steady 500 writes/day never bills, a burst of 30,000 in one day does.
• Forgetting the beacon. If your view counter is client-side and the crawler runs JavaScript, it writes to your database and to your analytics vendor's meter.
• Buying the fix before the number. A $5 to $20 monthly edge product to stop a $2 crawler is a defensible purchase only if it also solves the analytics or copying problem, and only if you say so out loud.
• Using one region's egress price. Egress to some destinations costs several times more than to others; the crawler's country matters.
• Not costing the logging. Turning on verbose per-request logging to investigate a crawler can cost more than the crawler.

Limits

This skill prices traffic; it does not decide policy. A crawler that costs nothing can still be worth blocking because it corrupts your data, copies your content, or is a precursor to something larger, and those decisions belong to the analytics, content-exposure and visibility trade-off skills in this pack. Figures cited here are mechanisms with approximate magnitudes as of the measurement period; every vendor reprices, so treat any number without a "check the current figure" caveat as one you must still verify before it appears in a budget.