You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have profiled crawlers from nginx logs with `awk`, from Cloud Logging with SQL, and from CloudFront logs with Athena, and you have written crawlers and then found your own signature in someone else's logs. You reason from request timing, header sets and network ownership, and you verify a claimed identity before you act on it.

## Key Points

3. **Order of pages.** Alphabetical, sitemap order, or ascending ID is a crawl. Human paths follow links.
4. **Did it ever fetch `/robots.txt`?** Declared crawlers do, usually first and then daily. Undeclared ones typically never do.
5. **Referer.** Absent on every request is consistent with automation; present and internal is consistent with clicking.
6. **Cookies.** Does the client ever send back a cookie you set? A fresh profile per page never does.
7. **Hour-of-day.** Flat across 24 hours is a machine; a sharp diurnal curve is a population.
8. **Status codes.** A client that keeps requesting after a `429` or `403` is ignoring you deliberately.
- **bingbot**: reverse name ends in `search.msn.com`; JSON ranges at `www.bing.com/toolbox/bingbot.json`.
- **Applebot**: reverse name ends in `applebot.apple.com`.
- **DuckDuckBot**: DuckDuckGo publishes an IP list on its help pages.
- **GPTBot, ChatGPT-User, OAI-SearchBot**: OpenAI publishes JSON range lists at `openai.com/gptbot.json`, `openai.com/chatgpt-user.json` and `openai.com/searchbot.json`.
- **Other AI vendors**: check the vendor's crawler documentation for a published list; if none exists, the token is unverifiable and gets no exemption.
- **Yandex**: reverse name in `yandex.ru`, `yandex.net` or `yandex.com` with forward confirmation.

## Quick Example

```bash
jq -r 'select(.request.uri | test("^/skills/")) | "\(.request.remote_ip)\t\(.request.headers."User-Agent"[0])"' access.log \
  | sort | uniq -c | sort -rn | head -20
```

```text
resource.type="cloud_run_revision"
logName:"requests"
httpRequest.requestUrl:"/skills/"
httpRequest.userAgent:"Chrome/"
NOT httpRequest.userAgent:("Googlebot" OR "bingbot")
```

Access Log Forensics for Bots

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade. You have profiled crawlers from nginx logs with awk, from Cloud Logging with SQL, and from CloudFront logs with Athena, and you have written crawlers and then found your own signature in someone else's logs. You reason from request timing, header sets and network ownership, and you verify a claimed identity before you act on it.

Core Principle: Logs Show Behaviour; Behaviour Is Hard to Fake at Scale

A User-Agent is a string the client chose. An IP is where the packets came from. Timing, request ordering, what gets fetched after the HTML, and what never gets fetched are things the client did. Profile the client by what it did, then use IP ownership to say who could have done it, and use the vendor's published verification to decide whether a claimed crawler identity is real. Never start from the User-Agent and never finish without the timing.

Where the Fields Are

Platform	Where	Client IP	User-Agent	Bytes	Latency
Cloud Run request logs	Cloud Logging, log name ending /requests, resource.type="cloud_run_revision"	httpRequest.remoteIp	httpRequest.userAgent	httpRequest.responseSize	httpRequest.latency
Cloud Logging via Log Analytics or BigQuery sink	Linked dataset, _AllLogs view	http_request.remote_ip	http_request.user_agent	http_request.response_size	http_request.latency
Vercel	Runtime logs (short retention by plan) or a log drain in JSON	proxy.clientIp	proxy.userAgent	proxy.responseByteSize	proxy timing fields vary; confirm names against your drain's sample payload
nginx combined	access.log	$remote_addr (field 1)	quoted field 6 with -F'"'	$body_bytes_sent (field 10)	needs $request_time in a custom format
Caddy JSON	access.log	.request.remote_ip and, behind a trusted proxy, .request.client_ip	.request.headers."User-Agent"[0]	.size	.duration
CloudFront standard logs	S3, tab-separated with #Fields: header	c-ip	cs(User-Agent) (URL-encoded)	sc-bytes	time-taken

Behind a proxy or load balancer, the IP field you want is the client hop of X-Forwarded-For, not the proxy's address; on Cloud Run remoteIp is already the client. Log Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site and Accept-Language yourself; no platform logs them by default.

Grouping: IP, /24, ASN, User-Agent

nginx combined, last 24 hours already filtered:

# top client IPs
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20
# top /24s (IPv4)
awk -F'[ .]' '{print $1"."$2"."$3".0/24"}' access.log | sort | uniq -c | sort -rn | head
# top User-Agents
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
# HTML navigations only, by IP
awk '$7 !~ /\.(js|css|png|jpg|webp|svg|woff2?|ico|json)(\?|$)/ {print $1}' access.log | sort | uniq -c | sort -rn | head

Caddy JSON with jq:

jq -r 'select(.request.uri | test("^/skills/")) | "\(.request.remote_ip)\t\(.request.headers."User-Agent"[0])"' access.log \
  | sort | uniq -c | sort -rn | head -20

Cloud Logging, Logs Explorer (filter only; it does not aggregate):

resource.type="cloud_run_revision"
logName:"requests"
httpRequest.requestUrl:"/skills/"
httpRequest.userAgent:"Chrome/"
NOT httpRequest.userAgent:("Googlebot" OR "bingbot")

Aggregation needs Log Analytics (SQL over the log bucket's linked dataset) or a BigQuery sink:

SELECT http_request.remote_ip AS ip,
       NET.IP_TO_STRING(NET.IP_TRUNC(NET.SAFE_IP_FROM_STRING(http_request.remote_ip), 24)) AS slash24,
       http_request.user_agent AS ua,
       COUNT(*) AS hits,
       COUNTIF(REGEXP_CONTAINS(http_request.request_url, r'/skills/')) AS html_hits,
       SUM(http_request.response_size) AS bytes,
       MIN(timestamp) AS first_seen, MAX(timestamp) AS last_seen
FROM `project.region.bucket._AllLogs`
WHERE resource.type = 'cloud_run_revision'
  AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY 1,2,3 ORDER BY hits DESC LIMIT 50;

CloudFront via Athena, using the column names from the standard-logs table DDL in the AWS documentation (request_ip is c-ip, user_agent is cs(User-Agent), bytes is sc-bytes, uri is cs-uri-stem):

SELECT request_ip, user_agent, COUNT(*) AS hits, SUM(bytes) AS bytes
FROM cloudfront_logs
WHERE date >= current_date - interval '7' day AND uri LIKE '/skills/%'
GROUP BY 1,2 ORDER BY hits DESC LIMIT 50;

Profiling a Candidate

Once an IP or /24 stands out, pull its full request list ordered by time and answer:

1. Inter-request spacing. A person produces clusters (a page plus its assets within a second, then a pause of seconds to minutes). A crawler produces a metronome. Compute the distribution of gaps between HTML requests; a standard deviation near zero, or a fixed sleep with uniform jitter, is automation.
2. What follows the HTML. A real browser fetches the stylesheet, scripts and fonts once, then reuses the cache. A fresh-profile headless browser fetches all of it every page. In the measured case: ~22 KB HTML then ~308 KB of assets on every visit, for 90 days, from one address. A curl scraper fetches HTML and nothing else.
3. Order of pages. Alphabetical, sitemap order, or ascending ID is a crawl. Human paths follow links.
4. Did it ever fetch /robots.txt? Declared crawlers do, usually first and then daily. Undeclared ones typically never do.
5. Referer. Absent on every request is consistent with automation; present and internal is consistent with clicking.
6. Cookies. Does the client ever send back a cookie you set? A fresh profile per page never does.
7. Hour-of-day. Flat across 24 hours is a machine; a sharp diurnal curve is a population.
8. Status codes. A client that keeps requesting after a 429 or 403 is ignoring you deliberately.

Browser-Owned Headers

Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site and Sec-Fetch-User are set by the browser engine and cannot be altered by page script. A top-level navigation is Sec-Fetch-Dest: document, Sec-Fetch-Mode: navigate, Sec-Fetch-Site: none (typed or scripted) or same-origin / cross-site (from a link). A fetch() is dest: empty, mode: cors or same-origin. Assets carry image, script, style, font.

What they tell you: the true request type, which lets you count navigations and ignore prefetches. What they do not tell you: whether a person is present. A headless Chrome sends exactly the headers a person's Chrome would for the same navigation, because it is Chrome. A client with no Sec-Fetch-* headers at all is not a modern browser: a script, a fetch library, or a very old engine.

Related signals worth logging and comparing against your known-human baseline rather than trusting outright: Accept-Language (automation frameworks default to en-US,en;q=0.9 regardless of the claimed country), the client-hint headers Sec-CH-UA, Sec-CH-UA-Platform and Sec-CH-UA-Mobile (consistent with the User-Agent in a real browser, sometimes not in a spoofed one), Upgrade-Insecure-Requests, and whether the client ever requests things a real session does after load: prefetches of visible links, the favicon, a second page.

Header Order and TLS Fingerprints

• Header order is preserved in Node's req.rawHeaders and in nginx's $http_* only per header, not as a sequence; WHATWG Headers objects (Fetch API, Next.js middleware) sort on iteration and lose order. Real Chrome sends a stable order for a navigation; python-requests, Go's net/http and curl each have their own. Log rawHeaders names at the origin if you need this signal.
• JA3 hashes the TLS ClientHello (version, cipher suites, extensions, curves, point formats). JA4 (2023) is a readable successor with variants for HTTP (JA4H) and others. They identify the TLS library, not the person: curl and requests look nothing like Chrome; headless Chrome looks exactly like Chrome. They are computed where TLS terminates, which on serverless is the platform's edge, so you only see them if the edge exposes them: Cloudflare (cf.bot_management.ja3_hash, cf.bot_management.ja4, plan-dependent), Vercel Firewall (JA4 as a rule condition), AWS WAF (JA3 and JA4 fingerprint match statements). On Cloud Run behind Google's front end you do not see them.

Conclusion for the headless-Chrome case: fingerprints exonerate curl-style scrapers quickly and say nothing about a real browser engine driven by a script. Behaviour (timing, fresh profiles, no cookies, no robots.txt fetch) is what identifies it.

Verifying a Claimed Crawler

Never trust the token. Verify:

# reverse DNS then forward confirmation (Googlebot)
host 66.249.66.1            # expect ...crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com   # expect the same IP back

• Googlebot: reverse name ends in googlebot.com or google.com; user-triggered Google fetchers resolve to googleusercontent.com. Google also publishes JSON range lists (googlebot.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json) under developers.google.com/search/apis/ipranges/.
• bingbot: reverse name ends in search.msn.com; JSON ranges at www.bing.com/toolbox/bingbot.json.
• Applebot: reverse name ends in applebot.apple.com.
• DuckDuckBot: DuckDuckGo publishes an IP list on its help pages.
• GPTBot, ChatGPT-User, OAI-SearchBot: OpenAI publishes JSON range lists at openai.com/gptbot.json, openai.com/chatgpt-user.json and openai.com/searchbot.json.
• Other AI vendors: check the vendor's crawler documentation for a published list; if none exists, the token is unverifiable and gets no exemption.
• Yandex: reverse name in yandex.ru, yandex.net or yandex.com with forward confirmation.

A request carrying Googlebot in its User-Agent from an IP that fails both checks is a scraper wearing a costume; treat it as an undeclared client.

Who Owns the Address

whois 203.0.113.7                                   # RIR record: netname, org, abuse contact
curl -sL https://rdap.org/ip/203.0.113.7 | jq '.name, .entities[].vcardArray'   # RDAP, JSON, redirects to the right RIR
whois -h whois.cymru.com " -v 203.0.113.7"          # Team Cymru: ASN, prefix, AS name

Hosting and cloud ASNs (the large clouds, VPS providers, "server" in the AS name) are where crawlers live; residential and mobile ASNs are where people live, and where residential-proxy scrapers hide. An offline ASN database (MaxMind GeoLite2-ASN, free with an account) lets you enrich a whole log in one pass without hitting a lookup service per line. Country from a GeoIP database is coarse and lags; use it as a correlate.

Reconstructing Sessions

Sort by (ip, user_agent, timestamp), split on a gap over 30 minutes, and compute per session: pages, assets, duration, first and last URL, whether a cookie ever came back. A stateless crawler produces thousands of one-page sessions from one IP; a person on that IP produces a handful of multi-page ones. If the crawler shares an ISP with humans (a NAT), the cookie split is what separates them. In SQL, LAG(timestamp) OVER (PARTITION BY ip, ua ORDER BY timestamp) gives the gap; SUM(CASE WHEN gap > 1800 THEN 1 ELSE 0 END) OVER (...) numbers the sessions.

The Cost of Looking

• Cloud Logging bills per GiB ingested beyond a monthly free allowance (historically 50 GiB per project) with a default retention that is free; Cloud Run request logs are roughly 1 to 2 KB each, so a million requests is one or two GiB. Custom per-request log lines with headers double that. Use exclusion filters for asset requests, or a sampling clause like sample(insertId, 0.1) in the sink filter when you only need proportions.
• Log Analytics and BigQuery charge per bytes scanned; partition by day and select the columns you need.
• Vercel runtime log retention is short and plan-dependent; a log drain to a third party is where the third party's per-GB price applies.
• CloudFront standard logs are free to produce; you pay S3 storage and Athena per TB scanned.
• Turn verbose logging on for the investigation window and off afterwards; a permanent header-level log of every asset request can cost more than the crawler you are hunting.

Checklist

• Grouped by IP, /24, ASN and User-Agent over at least 7 days, HTML separated from assets.
• Candidate profiled by inter-request gap, asset pattern, page order, robots.txt fetch, cookies, hour-of-day, and behaviour after 429.
• Claimed crawler tokens verified by reverse DNS with forward confirmation or published ranges.
• Ownership pulled via RDAP or whois with abuse contact recorded.
• Findings written up with the query, the period, and the numbers, so the conclusion is reproducible.
• Logging cost for the investigation estimated and verbose logging switched back off.

Common Mistakes

• Reading the leftmost X-Forwarded-For address, which the client controls.
• Blocking an ASN because one crawler lives there and taking a cloud VPN's users with it.
• Treating a matching JA3 as proof of a human; it is proof of a browser engine.
• Using GeoIP country as identity.
• Forgetting that the platform edge, not your origin, terminates TLS and owns the fingerprint.
• Paying for six months of verbose logs to answer a one-week question.

Limits

Logs establish what a client did and which network it came from. They do not establish intent, and they cannot verify a crawler whose operator publishes no ranges. A sufficiently careful client with residential proxies, real browser engines, cookie persistence and human-like pacing is indistinguishable in logs from a small population of readers; when you reach that point the questions become about cost, content exposure and policy, which are the other skills in this pack.