This is the method, its failure modes, and an honest account of what it costs to keep running. The goal is that you finish this page able to decide whether to build it or buy it — including deciding not to.
Everything here concerns public profiles, read logged out: the same page you see in a private browser window with no account. Nothing below involves credentials, private accounts, or content behind a login, and none of it should be used to get at those.
What "public" means, precisely
Open a private window, go to a public profile, and look at what renders before any interaction. Handle, display name, bio, profile picture, follower and following counts, post count, and the grid of recent posts with their thumbnails and captions.
That is the surface. It is genuinely public in the sense that matters technically — no account required — and it is what every vendor in this market is selling you, whatever the marketing copy implies.
Two things are not on that surface, and no amount of cleverness changes it: anything from a private account, and anything the platform only computes for the account owner. Reach, impressions, and audience demographics are owner-only. If a vendor offers you those for arbitrary profiles, that is worth a hard question.
The method, in principle
The page you get logged out is server-rendered with the profile data embedded in it. Fetching that page and reading the embedded data is the whole technique. There is no secret endpoint.
The three practical parts are:
1. request the public page with a plausible browser identity
2. locate the embedded JSON payload
3. read the fields you need, defensively
Step 3 is where the money is lost, and I will come back to it.
I am deliberately not publishing a copy-paste scraper with current selectors. Not out of coyness — it is that a working parser is a maintenance commitment, and a snippet frozen in a blog post is the single worst way to hold one. The selectors move. What follows is the part that stays true.
Why your server fails and your laptop works
This is the first thing that surprises people, and it is not subtle.
Datacentre IP ranges get rate limited first and hardest. Your laptop, on a home connection, looks like a person. An AWS or Hetzner box does not, and the platform treats it accordingly. The same code, byte for byte, succeeds locally and fails in production, which sends people hunting for a bug in their parser that does not exist.
The workaround the whole industry uses is residential proxy: traffic routed through consumer connections, priced per gigabyte. This is why per-GB pricing matters to anyone doing this at volume, and why Bright Data's pricing page is worth reading even if you never buy from them.
It is also the honest reason self-collection is not free. You replace a per-call fee with a bandwidth bill, an egress cost, and someone's afternoon when it breaks. It is roughly an order of magnitude cheaper per call. It is not zero, and anyone telling you it is has not run it.
The failure that does not throw
Here is the one worth the whole article.
Three ways a scrape fails, in ascending order of expense:
It errors. A 429, a 403, a timeout. This is the good case: it is loud, you retry with backoff, your monitoring notices.
It returns a page that is not the profile. A consent wall, a login interstitial, a challenge page. Status 200, HTML in the body, and your parser finds none of its fields. Annoying, but detectable — everything comes back empty at once, which is a pattern you can alarm on.
It returns the real page, and one field moved. Status 200. Nineteen fields parse correctly. The twentieth is now nested one level deeper, and your code writes null.
Nothing errors. Nothing looks wrong on a dashboard. Your row count climbs on schedule. And a week later you have seven days of history where follower count is empty, which you cannot backfill, because the past was not photographed.
We have written before about why an archive cannot be reconstructed after the fact, and this is the mechanism by which people lose one without noticing.
The defence is not cleverness in the parser. It is a check on the result:
- If a field is required, assert it is present in a sample of every run.
- A run where a required field is null in 100% of rows is not a quiet day. It is a broken parser, and it should page someone.
- Store the raw response for a short window. When the shape changes, you want yesterday's body to diff against — reconstructing it from memory is not possible.
That last one has paid for itself more than once. Debugging a parser against a response you no longer have is guesswork with a deadline.
Rate limits, and the arithmetic nobody does
The instinct is to hammer it until it breaks and then back off. Do the arithmetic first instead, because it usually changes the plan.
The questions that matter: how many profiles, how often, and how fresh does each one really need to be? A thousand profiles refreshed daily is a very different system from a thousand refreshed hourly — twenty-four times the traffic, twenty-four times the bandwidth, and twenty-four times the chance of tripping something.
Most requirements written as "real time" turn out to be "within a day" when you ask what decision the number feeds. That single question has cut more scraping costs than any optimisation I know of.
When you do hit limits: back off exponentially, add jitter so your retries do not synchronise into a second spike, and treat a block as a signal to slow the whole system rather than to retry that one call harder.
The legal part, which is not optional
Reading a public page is not a criminal matter, and Meta v. Bright Data (2024) found that logged-off collection of public data is not bound by terms you never accepted. That is a genuinely helpful precedent and it is narrower than people quote it as being.
What it does not do is exempt you from data protection law. Public data is still personal data under GDPR and LGPD. A follower count attached to a named human is personal data whether you found it behind a login or on an open page, and the moment you store it, you are the controller of it.
This is the part that vendors leave off the pricing page, and it is worth understanding properly before you store anything.
Build or buy
Honestly, from someone who chose to build:
Buy if this is a feature rather than your product, if you need it working this month, or if you do not want to own a parser that breaks on someone else's schedule. Several vendors do this competently and they are compared here.
Build if the per-call cost matters at your volume, if you need collection nobody can change without telling you, or if your vendor would be your competitor. That last one is a real consideration in this market and it is why we made the choice we made.
Neither, if what you actually need is to know whether something is growing. That is a question about change over time, and it is answered by an archive rather than by a scraper — which is the other product here, and it is live today.
Where we stand
The route we chose for ourselves is the one described above: public pages, logged out, our own client, no reseller in between. It is built and measured. It is not running.
That distinction matters here more than anywhere, because of the section above. What we have measured, we measured on a development machine over a home connection — by our own argument, the easy case. The proxy is not contracted, the collector is not deployed, and nothing is fetching on a schedule. Numbers taken from a laptop are not numbers from a datacentre, and ours have not been taken from one yet.
The position, including the parts that are uncomfortable, is written out in full on the social data page.
No social endpoint is live yet. When one answers, it appears with a real response and the cost per thousand calls next to it. Not before.
Written 7 September 2026. Platform behaviour, page structure and proxy pricing all drift — treat the mechanics above as a map rather than a specification, and verify against the live page before you commit engineering time.