The scraper fix that silently returns nothing

Your scraper broke. You found a fix. The requests come back 200, the job runs on schedule, the row count climbs.

You should check what is in those rows.

The failures that wake you up are the cheap ones. A 403 is loud, a timeout is loud, a 500 pages someone. The failure that costs real money is the one where the request succeeds and the content is wrong — because nothing errors, nothing alerts, and the damage accumulates at exactly the rate your job runs.

Here are three of them, measured on 7 September 2026, from a residential connection, logged out. All three return HTTP 200.

Case 1: the fix that is not a fix

When Reddit closed its unauthenticated .json endpoints, the obvious workaround was to point at old.reddit.com instead. It answers.

The second row is the trap. It is the only Reddit surface that passes a status check, and it does not return JSON — it returns a rendered web page.

If your code is if (res.ok) { parse(await res.json()) }, you now throw on the parse and hopefully notice. If it is wrapped in a try/catch that logs and continues — and an enormous amount of production scraping code is — you write nothing and report success.

Case 2: real data, missing the field you came for

Reddit's RSS still works. It returns genuine Atom XML: 25 items, with titles, links, authors, timestamps and content. Everything looks healthy.

I checked the whole payload for metrics:

score          absent
ups            absent
num_comments   absent
upvote         absent
points         absent

The word "comments" does appear — inside post URLs, as a path segment. It is not a count.

So the feed tells you what was posted and cannot tell you what did well. If your pipeline reads item.score and writes it, you are writing null on every row, forever, and no retry or backoff will change it: the number is not in the payload.

This one is nastier than Case 1 because the response is legitimately valid. There is no malformed body to trip over. Your parser is working correctly on data that does not contain what you need.

Case 3: a profile that does not exist, returning a profile page

This is the one I would put on a poster.

I requested three Instagram profiles: two real, one a handle I made up by mashing the keyboard.

The handle that does not exist returned more bytes than the two that do. Its title is identical. I searched its body for "Page Not Found", "Sorry, this page" and "isn't available": zero matches.

Every signal a normal scraper checks — status code, response size, page title, presence of an error string — says this profile is fine. It is not a profile at all.

The mechanism is ordinary: the page is a JavaScript shell, and the profile data arrives afterwards through an internal call. The shell is nearly the same whoever you ask for. But the consequence is not ordinary. Feed a list of handles through a validator built on status codes and you will "confirm" every typo, every renamed account and every deleted user in it.

Why this class of bug is the expensive one

Three properties, and each one costs separately.

It does not surface. No exception, no alert, no red on a dashboard. The first person to notice is usually a customer, and what they noticed is that your numbers are wrong.

It corrupts silently at the rate of your schedule. An hourly job that started writing nulls on Tuesday has written 120 bad rows by Sunday, interleaved with good ones. Finding the boundary afterwards is archaeology.

Time-series data cannot be backfilled. This is the part people underestimate. If you were storing a follower count every hour and the field went null for six days, those six days are gone. Not "expensive to recover" — gone. Nobody photographed them, and no vendor sells you last Tuesday. It is the whole reason an archive has to start before you need it.

Compare that to a 403, which stops your job, wakes you up, and costs you the hours until you fix it. The loud failure is the cheap one.

The check that catches all three

The mistake underneath every case above is the same: treating the status code as a claim about the content. It is not. It is a claim about the transport.

What actually works is asserting on the shape of what came back, and doing it every run rather than at build time.

Assert a required field exists. Pick a field that must be present in a valid response. Check a sample of every run. In Case 2, a single assertion on score catches the problem on day one instead of month three.

Alarm on 100% null. This is the highest-value alert in scraping and almost nobody has it. A run where a required field is empty in every row is not a quiet day on the internet — it is a broken parser. Zero variance is the signal.

Check content type against expectation, not just status. In Case 1, one comparison — content-type says text/html, you asked for JSON — ends it immediately.

Validate identity, not just presence. In Case 3, the fix is asserting that the response actually contains the handle you asked for. If you requested @nasa and the body never mentions nasa, you did not get @nasa, whatever the status code says.

Keep raw responses for a short window. When the shape changes, you want yesterday's body to diff against. Reconstructing it from memory is guesswork, and the retention cost is trivial next to a week of bad rows.

Watch row-count variance, not just row count. Counts stay healthy in all three cases above. What changes is the distribution — fields going uniformly empty, sizes clustering suspiciously tight. Monitor the shape, not the volume.

The one-line version

A 200 means the server answered. It does not mean it answered your question.

Every case above is that sentence, played out in a different way. Reddit answered with a web page. RSS answered with real data missing the column. Instagram answered with a shell for an account that does not exist.

What this is really about

Anyone can write a scraper that works today. The work is the part after: noticing when it stops being right, given that it will not tell you.

That means assertions on content, alarms on zero variance, retained raw bodies, and someone who re-measures when a platform moves — because platforms move constantly now, and several of them moved this year in ways that broke everything downstream without a single error being raised.

If that sounds like more than you want to own, that is the honest reason this category of vendor exists. It is also the honest reason to ask any vendor how they detect it, and to be suspicious of one who has not thought about the question — because they are exposed to exactly the same three failures, on your behalf, and you will find out later than they do.

The route we picked is to collect from public pages ourselves, logged out, rather than through a reseller — designed and measured, and not yet running. The position and its trade-offs are on the social data page. No social endpoint of ours is live yet, and that page says so rather than listing what does not exist.

All measurements taken on 7 September 2026 from a residential connection, logged out, with a browser user-agent — primary measurement, reproducible with curl. Platform behaviour changes; re-measure before relying on any of it. That is the entire point of the article.