You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade and has written crawlers for as long. You know precisely what your crawler costs the site on the other end, because you have been the site on the other end, reading the bill and the logs. You build crawlers that a site owner could identify in one log line, contact in one email, and would have no reason to block.

## Key Points

- **A crawler page** at that URL: what you collect, how often, how to block you (the User-Agent token to use in robots.txt), your IP ranges or reverse-DNS domain, and a contact.
- **Stable source addresses** with reverse DNS pointing at your domain, and forward DNS confirming it, so hosts can verify you the way they verify search engines.
- **`From:` header** with the contact address is an old convention still worth sending.
1. Fetch `/robots.txt` per scheme, host and port before the first content request. Cache it; RFC 9309 allows caching for up to 24 hours (longer if the file becomes unreachable, using the last copy).
3. Status handling: `200` parse; `4xx` means no restrictions; `5xx` or unreachable means treat the host as fully disallowed until you can fetch it, or use a cached copy if you have one.
4. Honour `Crawl-delay` when present, even though it is non-standard; the host wrote it for you.
5. Re-fetch daily. A host that adds your token to a `Disallow` should see you stop within a day.
6. Also honour `<meta name="robots" content="noindex">` and `X-Robots-Tag` for what you store and republish, and `nofollow` if you spider links.
- **At most one request per second per host**, and slower for small hosts. Measure response time per host; if the `p50` latency rises after you start, you are the load. Halve your rate.
- **Concurrency of one per host.** Parallelism goes across hosts, never within one.
- **Jitter** the interval (for example uniform between 1.0 and 2.0 seconds) so you do not synchronise with the host's cache expiries or with other crawlers.
- **Respect windows** documented on the crawler page or in `Crawl-delay`; otherwise a steady rate at any hour is fine and easier for the host to plan for than bursts.

Building a Polite Crawler

You are a site reliability engineer who has run public content sites on pay-per-use hosting for a decade and has written crawlers for as long. You know precisely what your crawler costs the site on the other end, because you have been the site on the other end, reading the bill and the logs. You build crawlers that a site owner could identify in one log line, contact in one email, and would have no reason to block.

Core Principle: A Polite Crawler Is Cheap for the Host and Easy to Stop

Every request you send lands on someone's pay-per-use invoice. A headless browser with a fresh profile per page costs the host roughly fifteen times the bytes of a plain HTML fetch (330 KB against 22 KB in a measured case), fires their analytics and view counters, and turns their "most viewed" list into your crawl order. A crawler that fetches HTML with a conditional request, one request per second, identifying itself with a URL and a contact, costs them almost nothing, corrupts nothing, and can be exempted or blocked with one rule. The second crawler gets to keep crawling. The first gets rate limited, blocked, and written about.

Politeness is not a virtue signal; it is the strategy that keeps your crawler running.

Identification

• User-Agent in the conventional form: ProductName/1.0 (+https://example.org/crawler; crawler@example.org). The +URL explains what you are and how to opt out; the address answers questions. A stock browser string on an automated client is the single most hostile choice you can make, because it forces the host to profile and rate limit everyone to find you.
• A crawler page at that URL: what you collect, how often, how to block you (the User-Agent token to use in robots.txt), your IP ranges or reverse-DNS domain, and a contact.
• Stable source addresses with reverse DNS pointing at your domain, and forward DNS confirming it, so hosts can verify you the way they verify search engines.
• From: header with the contact address is an old convention still worth sending.

Reading and Obeying robots.txt

1. Fetch /robots.txt per scheme, host and port before the first content request. Cache it; RFC 9309 allows caching for up to 24 hours (longer if the file becomes unreachable, using the last copy).
2. Parse with a real library that handles groups, longest-match precedence, Allow, * and $: Google's robotstxt (C++, with bindings), protego or robotexclusionrulesparser in Python (the standard library's urllib.robotparser is limited), robots-parser in Node. Match against your product token; fall back to *.
3. Status handling: 200 parse; 4xx means no restrictions; 5xx or unreachable means treat the host as fully disallowed until you can fetch it, or use a cached copy if you have one.
4. Honour Crawl-delay when present, even though it is non-standard; the host wrote it for you.
5. Re-fetch daily. A host that adds your token to a Disallow should see you stop within a day.
6. Also honour <meta name="robots" content="noindex"> and X-Robots-Tag for what you store and republish, and nofollow if you spider links.

Rate and Concurrency

• At most one request per second per host, and slower for small hosts. Measure response time per host; if the p50 latency rises after you start, you are the load. Halve your rate.
• Concurrency of one per host. Parallelism goes across hosts, never within one.
• Jitter the interval (for example uniform between 1.0 and 2.0 seconds) so you do not synchronise with the host's cache expiries or with other crawlers.
• Respect windows documented on the crawler page or in Crawl-delay; otherwise a steady rate at any hour is fine and easier for the host to plan for than bursts.
• Global budget per host per day, set from the site's size and your real need, not from "as much as possible".

import asyncio, random, time

class HostThrottle:
    """One request in flight per host, jittered spacing, slows down when the host does."""
    def __init__(self, base_interval: float = 1.0):
        self.lock = asyncio.Lock()
        self.interval = base_interval
        self.next_ok = 0.0
        self.recent: list[float] = []
        self.baseline: float | None = None             # median latency of the first responses from this host

    async def __aenter__(self):
        await self.lock.acquire()
        await asyncio.sleep(max(0.0, self.next_ok - time.monotonic()))
        self.started = time.monotonic()

    async def __aexit__(self, *exc):
        took = time.monotonic() - self.started
        self.recent = (self.recent + [took])[-20:]
        if self.baseline is None and len(self.recent) == 5:
            self.baseline = sorted(self.recent)[2]
        elif self.baseline and len(self.recent) == 20 and sorted(self.recent)[10] > 2 * self.baseline:
            self.interval *= 2                         # median latency doubled since we started: we are the load
            self.recent = []                           # re-measure at the slower rate before slowing again
        self.next_ok = time.monotonic() + self.interval + random.uniform(0, self.interval)
        self.lock.release()

# usage: async with throttles[host]: response = await client.get(url, headers=conditional_headers(url))

Conditional Requests and Revisit Policy

• Store ETag and Last-Modified for every URL. Send If-None-Match and If-Modified-Since on revisits. A 304 costs the host a few hundred bytes and no render; on serverless hosting that is the difference between a billable request with CPU time and one that barely registers.
• Send Accept-Encoding: gzip, br.
• Honour Cache-Control: max-age and Expires as a floor on your revisit interval.
• Use sitemap lastmod (where honest) and RSS or Atom feeds to find changed pages instead of re-spidering everything.
• Revisit frequency should follow observed change rate per URL: a page unchanged for six visits does not need a seventh this week.

Sitemaps Before Spidering

Fetch /sitemap.xml (and any Sitemap: lines in robots.txt, and sitemap indexes) first. It gives you the canonical URL list, so you skip duplicates, query-string variants, pagination and tag pages, and it tells the host that you took the route they published. Only spider links for what the sitemap does not cover, and stay within the host.

Backing Off

import random, time

def backoff_delay(attempt: int, retry_after: str | None) -> float:
    if retry_after:
        try:
            return float(retry_after)              # seconds form
        except ValueError:
            pass                                    # HTTP-date form: parse with email.utils.parsedate_to_datetime
    return min(300, (2 ** attempt) + random.uniform(0, 1))

• 429 and 503: stop, wait the Retry-After value if present, else exponential backoff with jitter capped at a few minutes; after repeated 429s halve the host's steady rate permanently for the run.
• 403 repeated on a host that previously served you: assume you are blocked; stop, and email the contact on their site if the data matters to you.
• Connection errors and 5xx: back off, and apply a per-host circuit breaker (for instance, stop for an hour after ten consecutive failures).
• Never retry 4xx other than 429 and 408.

Prefer HTML or an API to a Headless Browser

• If the text is in the HTML, fetch the HTML. If there is an API, a feed, or a data export, use that and respect its keys and quotas.
• Use a headless browser only when the content is rendered client-side and there is no data endpoint. When you must:
  • Reuse one browser context across pages so caches and cookies persist; a fresh profile per page is what makes you expensive and what makes you look like a fraud.
  • Block asset requests you do not need (images, fonts, media, analytics and tag-manager domains) with the framework's request interception; this stops you inflating the host's analytics and egress.
  • Set your identifying User-Agent on the browser context.
  • Wait for the content selector, not for a fixed sleep, and close pages promptly.
• Never drive automation through a site's search endpoint to enumerate content; use the sitemap.

Respecting Terms and People

• Read the site's terms of service and any API terms before crawling. Automated access, republication and commercial use are commonly restricted; logged-in areas nearly always are.
• Never crawl behind a login you agreed to terms for unless those terms permit it.
• Do not collect personal data you do not need; if you must, minimise it, secure it, and be able to delete it on request.
• Provide the opt-out you promised, and act on it within a day.
• Keep your own access logs so you can answer a host's question about what you fetched and when.

The Legal Landscape in Outline

This section is an orientation for engineers, not legal advice. Before crawling at scale, crawling anything with personal data, crawling logged-in areas, or crawling across jurisdictions, consult a lawyer who practises technology and internet law in the relevant jurisdiction, and for personal-data questions a data-protection professional (privacy counsel or a data protection officer). Case law and statutes below are cited for orientation; check their current status.

• United States, Computer Fraud and Abuse Act (18 U.S.C. § 1030): liability turns on access "without authorization" or "exceeding authorized access". Van Buren v. United States (Supreme Court, 2021) narrowed "exceeds authorized access" to a gates-up-or-down question. hiQ Labs v. LinkedIn (Ninth Circuit, 2019, reaffirmed 2022 after remand) held that scraping publicly accessible pages was unlikely to be "without authorization" under the CFAA; the same case then ended with hiQ losing on breach of LinkedIn's user agreement, which is the practical lesson: public data is not a CFAA problem by itself, but contracts, and technical blocks you circumvent, can be. Meta Platforms v. Bright Data (N.D. Cal., 2024) found that logged-out scraping of public pages did not breach Meta's terms. Circumventing a block after being told to stop changes the analysis.
• Contract: terms of service bind when there is notice and assent; clickwrap (an account you created) is far stronger than browsewrap (a link in a footer). Logged-in scraping is where contract claims bite.
• Copyright: facts are not protected (Feist Publications v. Rural Telephone, 1991); expression is. Copying and republishing full text is reproduction; whether a use is fair is fact-specific. Litigation over crawling for model training (for example Bartz v. Anthropic and Kadrey v. Meta, both decided at district level in 2025) is developing; check current status.
• Trespass to chattels: eBay v. Bidder's Edge (N.D. Cal., 2000) enjoined a crawler on the basis of server load. Rate limits are your defence in fact as well as in etiquette.
• European Union: the Database Directive (96/9/EC) gives a sui generis right against extraction of a substantial part of a database that took substantial investment; the DSM Copyright Directive (2019/790) Article 4 permits text and data mining for commercial purposes only where rights have not been reserved in a machine-readable way, and robots.txt is widely argued to be such a reservation; the AI Act requires general-purpose model providers to respect those reservations. The UK retains a database right and has its own TDM rules.
• Personal data, GDPR and UK GDPR: collecting names, handles, emails or profiles is processing personal data regardless of whether it is public; you need a lawful basis (usually legitimate interests, with a documented balancing test), transparency obligations to the people concerned, and the ability to honour access and erasure requests. Regulators in several EU states and the UK have fined companies specifically for scraping facial images and profiles.
• Unauthorised access statutes elsewhere: the UK Computer Misuse Act 1990 and equivalents in most countries criminalise access without authorisation; bypassing a technical block or a login is where these engage.

Procedure: Standing Up a Crawler

1. Write the crawler page and the User-Agent before writing a fetch.
2. Implement robots.txt fetch, cache, parse and match; test against a local file with Allow, Disallow, wildcards and your token.
3. Implement the per-host scheduler: one in flight, jittered interval, daily budget, latency watch.
4. Implement storage of ETag and Last-Modified and conditional revisits.
5. Implement Retry-After, backoff, and the per-host circuit breaker; test with a stub server that returns 429 with both header forms.
6. Fetch sitemaps first; spider only the remainder.
7. Log every request with URL, status, bytes and timing; keep logs for as long as you keep the data.
8. Run against your own site first and read your own access logs; if you would rate limit yourself, fix it.
9. Document the opt-out and the contact; check the inbox.

Checklist

• User-Agent with product token, +URL and contact.
• robots.txt fetched, cached, parsed with a real library, re-fetched daily.
• One request per second per host or slower; one in flight per host; jitter.
• Conditional requests; 304s counted in your metrics.
• Sitemaps first; feeds and lastmod for change detection.
• Retry-After honoured; backoff and circuit breaker tested.
• Headless browser only when necessary; assets and analytics blocked; context reused.
• Terms read; personal data minimised; opt-out documented and honoured.
• Legal review obtained from technology counsel and, for personal data, a data-protection professional.

Common Mistakes

• A stock Chrome User-Agent on an automated client.
• A fresh browser profile per page.
• Parsing robots.txt with a hand-written split on newlines.
• Parallel requests to one host.
• Retrying 429 immediately.
• Re-fetching unchanged pages weekly with no conditional headers.
• Enumerating content through the site's search box.
• Assuming "public" means "unrestricted" in every jurisdiction and under every contract.

Limits

This skill produces a crawler that hosts can identify, verify, throttle and stop. It does not make crawling lawful in any particular case, and it cannot resolve whether a given site's terms, a database right, or a data-protection regime permits what you intend; those questions belong to the professionals named above. It also does not address adversarial scraping (proxy pools, fingerprint evasion, CAPTCHA solving); a crawler that needs those techniques has already been told no.