I run an hourly collector that reads public developer platforms and writes what it finds into an archive. Which platforms it should read is data, not code: one row per platform in a Postgres table, with a boolean column ativo meaning "collecting right now". I counted the collector's own ledger against that table. They did not agree, and the ledger had no way to say so.
All numbers below were read on 2026-09-10, between 18:00 and 18:10 UTC, from the production database, and re-counted against the same rows on 2026-09-22 before publishing.
The count
The ledger (coleta) holds one row per run × platform × topic. 3,038 rows. Three states:
ok— 2,413vazio(nothing came back) — 619erro— 6
Six errors in 3,038 attempts. 99.8% not-error, on a system where I already knew three sources had never collected anything.
Here is where the 619 empties came from. The ativo column is as it read on 2026-09-10:
| platform | ativo |
empty rows |
|---|---|---|
| false | 162 | |
| false | 162 | |
| tiktok | false | 162 |
| bluesky | false | 131 |
| lobsters | true | 2 |
Those 619 rows cover three different situations:
- 486 belong to reddit, instagram and tiktok — three rows inserted with
ativo = falseand never switched on. I checked every migration in the project's history: the only platform ever flipped totrueby one is YouTube. - 131 belong to Bluesky, which was not idle at all. In the same window it also produced 25
okrows and 6 explicit block errors, so it was being called and refused. - 2 belong to Lobsters, a live source that genuinely had nothing new in the 07:00 UTC window on 8 September.
Off, blocked, and quiet. All three are written as the same string. In all 619 vazio rows, erro_tipo is NULL and erro_detalhe is NULL. There is no column that separates them.
42 hours of calling sources that were marked off
Reddit's row was created with ativo = false on 2026-09-06 at 19:07 UTC. TikTok and Instagram were created false at 22:18 UTC the same day. No later migration flips any of them to true.
The collector logged attempts against all three anyway: 162 each, 486 in total, across 42 distinct hourly windows, first at 2026-09-07 21:07:18 UTC and last at 2026-09-09 15:07:18 UTC. The arithmetic is exact and boring: 54 runs × 3 topics = 162 per platform.
So the table said off, the scheduler kept going, and the ledger recorded 486 rows that look identical to a quiet hour on a healthy source. The config and the code had diverged, and the log was the wrong instrument to notice it with — it recorded faithfully that nothing came back, which was true and useless.
The calls stop at 2026-09-09 15:07:18 UTC. I have not verified from the database what changed at that moment; there is no migration at that timestamp.
The one source that failed loudly
Bluesky is the interesting counterexample. It was genuinely being called, and it is the only platform in the entire ledger that ever produced an erro row — all six of them:
erro_tipo: bloqueio
erro_detalhe: tentativas_esgotadas apos 3: http 403; 10/10 termos falharam;
30 tentativas gastas
That row tells me what happened, how many terms failed, and how many retries it burned. Its 162 attempts split into 25 ok, 131 vazio, 6 erro — a source alternating between working and not, which is worse than one that is plainly down, because the ok windows are real and the archive keeps them.
It also cost the most. Total recorded duration across Bluesky's 162 attempts: 6,668,359 ms, about 1 hour 51 minutes of collector wall-clock, averaging 41 seconds per attempt — mostly waiting. Reddit, Instagram and TikTok together spent 148,724 ms across their 486 attempts, about 306 ms each (305, 305 and 307 respectively). The expensive failure and the cheap one are both vazio.
The alerting did not help, by design
I have an alert table for stopped sources. On 2026-09-10 it had zero rows. That is not a bug in the alerter: a gap is defined as "a pair that was producing and stopped", and a pair that never produced anything cannot stop. The definition is right. The consequence is that a source which never worked is invisible to it forever, and the three sources with the most empty rows in my ledger are exactly the three that never produced.
It is no longer empty. Re-reading the same table on 2026-09-22, it holds 8 incidents, none of them still open — every one opened by a mechanism built after this reading, none by the gap definition above.
What I am not claiming
- I did not measure Reddit, TikTok or Instagram. These are records from my scheduler, not statements about anyone's API. Nothing here says those platforms behaved badly.
- The ~306 ms average does not prove an HTTP request left the machine.
duracao_msis the recorded duration of the attempt. I cannot tell from this table whether a socket was opened, so I am not claiming wasted requests against a third party — only wasted scheduled work. - The ledger begins at 2026-09-07 21:07:18 UTC; the archive already holds rows from 2026-09-05 20:00 UTC. Attempts made before the ledger existed are not counted, so every count here is a floor, not a total.
- Bluesky's rows are still in the archive: 1,254 observations across 24 windows, 2026-09-06 19:00 to 2026-09-09 13:00 UTC. I did not check whether the read path filters inactive platforms, so I am not claiming they are served, only that they are stored.
ativohas no history column. I can say what it reads today and what every migration in the repository did to it; I cannot prove from the database alone what it read at the instant of any single attempt.
Reproduce it on your own collector
The whole finding is two queries. If your collector writes a per-attempt ledger:
-- 1. how many states does "nothing came back" actually cover?
select estado, count(*),
count(*) filter (where erro_tipo is not null) as com_tipo
from coleta group by 1 order by 2 desc;
-- 2. which of those empties belong to sources you switched off?
select c.plataforma, p.ativo, count(*) as vazios,
min(c.janela) as primeira, max(c.janela) as ultima
from coleta c join plataforma p on p.id = c.plataforma
where c.estado = 'vazio'
group by 1, 2 order by 3 desc;
If the second query returns rows with ativo = false, your scheduler and your configuration are two different lists, and your log cannot tell you which one is lying. The cheap fix is not a new alert. It is making the empty case carry a reason, so that "off by choice", "blocked", and "quiet hour" stop sharing a column.
Happy to share the rest of the ledger schema, or the per-platform duration numbers, if that is useful to anyone measuring the same thing.