The Crawler That Was 87% of Our Users

Friday, 06:40. The admin dashboard, third coffee. Top Countries: Singapore, 15,563 users. United States, 957. China, 530. Ireland, 58.

I looked at that list for a full week before the number bothered me. Singapore is a fine place. It has excellent broadband and a lot of people who write software. It does not have sixteen times more people reading a catalogue of agent skills than the United States does.

So I pulled the raw report instead of the pretty one. Thirty days, one country, every dimension I could think of.

Signal	Singapore	Everyone else
Users / sessions / pageviews	15,672 / 15,673 / 15,797	2,311 / 3,013 / 5,269
Pages per session	1.0	1.7
Engagement rate	3%	16% to 70% depending on the country
Average session	4.9 seconds	38 seconds to almost 10 minutes
Device, OS, browser, source	100% desktop, 97% Windows, Chrome, "(direct)"	mixed, like humans
New-user events	15,652	normal

Read the last row again. Fifteen thousand six hundred and fifty-two first visits out of fifteen thousand six hundred and seventy-two users. Every single hit was a brand-new person who had never been here before, arrived by typing the URL, looked at exactly one page for five seconds, and left forever.

That is not a country. That is one machine with a fresh browser profile for every page.

---

#Day 1, 07:15. What it is

Headless Chrome. Stock Windows user agent, so it looks like a person. It runs JavaScript, which is the part that matters, because Google Analytics only counts you if your JavaScript runs. GA's bot filter catches the honest crawlers, the ones that announce themselves. This one announces nothing.

It hit 4,930 distinct skill pages in thirty days. We have 6,159 skills in 447 packs. It had read roughly four-fifths of the catalogue, about three times over, on a ten-day cycle, at a steady four to five hundred pages a day, for at least ninety days. Median twenty pages an hour. Never more than fifty-six. The busiest hour the entire United States produced in two weeks was thirty-five.

Somebody is copying the catalogue. Slowly, politely, and in a way that no dashboard I own flagged as anything but growth.

I sat with that for a while. Then I did what you do, which is ask the question the accountant would ask.

#Day 1, 09:00. What it costs

Here is the honest number, and I want to be honest about it because the internet is full of people yelling about bot traffic without doing the arithmetic.

Per fresh visit, the crawler downloads about 22 KB of HTML and up to 308 KB of static assets, because a fresh profile has an empty cache. Each skill page renders server-side in about a quarter of a second of one vCPU. We run on Cloud Run with a single vCPU, scale to zero, CPU billed only while a request is in flight.

Line item	Monthly volume	Cost
CPU for ~15,800 page renders	~4,000 vCPU-seconds	inside the free tier
Requests, assets included	~250,000	inside the free tier
Egress to Asia, worst case, no asset cache	~5 GB	about sixty cents
Database writes from our own view beacon	~15,000	inside the free tier

Under two dollars a month. I could not even make it expensive. Blocking it with a managed bot product would cost more than the bot.

So why am I writing this at 09:00 with a cold coffee instead of closing the tab?

#Day 1, 09:20. What it actually costs

Because the bill was never the point. Here is what that crawler was doing to us that no invoice shows.

Every number I make decisions from was three-quarters machine. Eighty-seven percent of "users." Seventy-five percent of pageviews. The growth chart. The country list I used, two days earlier, to decide which country-specific skill packs to build next. I built a Singapore pack partly because of a bot.

"Most Viewed Skills" was ranked in crawl order. The crawler runs a real browser, so it fired the same view beacon a reader fires. Its views were perfectly uniform, fifteen to eighteen per page, which means the "most viewed" list was closer to a random number generator than to taste. And I had been reading that list as demand.

It has the full text of everything. Our skill pages ship the whole file in the HTML, because search engines need to read it. The "sign in to see the rest" gate is a blur applied in the browser. A crawler does not run the blur. Whoever this is has the catalogue, three times over, kept fresh.

I once watched a man spend forty minutes proving to a parking attendant that he had, in fact, paid. He was right. He also missed his flight. That is the bot-traffic argument in one image: you can win the "it's only two dollars" fight and still lose everything that mattered.

---

#Day 2, 02:10. The trap on the other side

Now the part I want you to hear if you run a small site, because this is where the little guy pays twice.

The obvious move is to hide. Block the crawlers. Put a challenge page in front of everything. Turn on the bot-fight toggle at the edge.

Here is what that costs you.

The declared crawlers are the ones that send you people. Googlebot, Bingbot, the AI search fetchers that cite you when someone asks a question. They identify themselves, they read robots.txt, and they are the only crawlers a robots.txt file can stop, because the polite ones are the only ones who read it. The one that copied my catalogue never fetched robots.txt in its life.

So blocking by robots.txt takes traffic away from you and does nothing to the machine that was actually the problem. A JavaScript challenge at the edge does stop the machine, and it also stops link previews, some search crawlers, the AI assistants that would have cited you, and every reader on a locked-down corporate network. And the edge products that do this well cost five dollars a month plus per-rule plus per-request, which, as established, is more than the crawler.

That is the whole asymmetry. If you are big, bot traffic is a rounding error and you have a team. If you are small and pay per request, the crawler's cost lands on your bill, hiding from it lands on your visibility, and the products sold to fix it cost more than the problem. Who's paying is the little guy, in one currency or the other.

The anchor sentence, then, plainly: you cannot buy your way out of this, and you cannot hide your way out of it. You can only measure your way out of it.

#Day 2, 03:30. What we did instead

Three things, none of which cost money, all of which came from the data rather than from fear. I wrote the whole method up as a pack while it was fresh, bot-traffic-and-crawler-defense-skills, so the next person does not have to spend a week staring at a country list.

1. Stop reading "users." GA already has a number a crawler cannot fake: the engaged session. Ten seconds on the page, or two pages, or a conversion. The crawler's engagement rate was three percent. Every human country was between sixteen and seventy. We switched the dashboard to rank countries by engaged sessions, and Singapore fell from first to a badge that says crawler. The exact dimensions to pull and the signature to look for are in bot-traffic-and-crawler-defense-skills/bot-traffic-signatures-in-analytics.

2. Make the view beacon wait. A view is a person who stayed. Ten seconds, tab visible, not an automated browser. The crawler leaves at five. "Most Viewed" started meaning something again by the next morning.

3. Rate-limit the page, not the API, and size it from data. Sixty skill pages an hour per address, two hundred a day. The crawler runs a median of twenty an hour and five hundred a day; a human does not read two hundred full skill files in a day. Search crawlers are exempt because robots.txt is the right tool for them. Every hit over the line gets a 429, a Retry-After, and a plain page that says what happened and where the API is.

This is the part that bit us on the way in, and it is why the method is a skill and not a tweet. The first version keyed on the framework's own prefetch headers, so that a reader scrolling the listing, which prefetches sixty pages, would not get counted. Then I tested it against the production build and the rule never fired. Next.js strips those headers before the middleware ever runs. The only signal the browser owns and a script cannot forge is Sec-Fetch-Dest. A navigation says document; the router's own prefetches and clicks say empty; a script with no Sec-Fetch-Dest at all is, by definition, a script.

// middleware: what counts toward the per-IP skill-page limit
export function isNavigation(headers: Headers): boolean {   const dest = headers.get('sec-fetch-dest');   if (dest) return dest === 'document'; // browser: navigations only   return true;                          // no Sec-Fetch-Dest = not a browser }

Proven the boring way: sixty-first page load from one address is a 429, a real Chrome scrolling the listing and clicking a result produced zero counted hits, a typed URL produced exactly one. The design, the sizing, and the pitfalls are in bot-traffic-and-crawler-defense-skills/web-rate-limiting-design, and the cost arithmetic that tells you whether you even have a bill problem is in bot-traffic-and-crawler-defense-skills/crawler-cost-accounting.

And every counted hit now logs the address and the user agent. By Monday I will know whose machine it is. That is the real deliverable. Not a wall. A name.

#Day 2, 04:05. What it is not

It is not a wall. Anyone willing to rotate addresses reads the site at leisure, because the text is in the HTML and the text is in the HTML because we want to be found. The wall would be serving the preview to anonymous visitors and the full file to people with a key, and that trades away search visibility for protection. That is a product decision. I have not made it. The trade-off itself, who to let in and what each group gives back, is bot-traffic-and-crawler-defense-skills/seo-and-ai-search-visibility-tradeoffs, and I would rather you make it with the numbers in front of you than with a toggle.

If you are on the other side of this, if you are the one writing the crawler, there is a file in the pack for you too, bot-traffic-and-crawler-defense-skills/building-a-polite-crawler. Announce yourself. Read robots.txt. One request a second. Back off on a 429. The site you are reading is being paid for by someone, and it is probably not who you think.

---

Pull the country report on your own site. Not the summary card. The raw one, with engagement rate and pages per session next to users. If one row has ten times the users and a tenth of the engagement of every other row, you have a tenant.

Then load the pack into whatever agent you use to run the place, and let it do the arithmetic you have been avoiding: skilldb.dev/skills.