How we cluster breaking news across hundreds of sources

The problem

On 28 February 2026, in the hour after the first reports of strikes on Iran, roughly a hundred news apps in more than twenty countries pushed a notification about it, in Swedish, Malayalam, French, Hebrew, German, English, Italian and Romanian. In our twelve-month dataset there were 15,942 stories that at least three apps pushed, 1,846 that ten or more pushed, and one that 121 did. The median gap between the first and second app on the same story is 16 minutes; in Sweden it is under four.

A monitoring deck that shows all of those as separate cards is useless in exactly the moments it is supposed to be useful. What a desk wants is one card that says "this story, 125 sources, first pushed by X at 06:16". And it wants that card while the story is still breaking, not after a batch job has run. That is the clustering problem, and this page is how we solve it: honestly, including the parts we are not proud of.

Design constraints

• Seconds, not minutes. A push arrives; within a few seconds it must either join an existing story or start a new one. Nothing can wait for "the batch".
• Every push gets compared. Hundreds of apps and robots produce a steady stream; whatever we do per push has to be cheap enough to do thousands of times a day, and whatever we do with a language model has to be gated so it does not run on every pair.
• Thirty-one languages. The same story arrives in Finnish and Portuguese within the same minute. Clustering has to be language-agnostic.
• Over-merging is worse than under-merging. Two cards for one story is annoying. One card for two stories hides news. Every threshold below is tuned with that asymmetry in mind. The failure modes we list are mostly the places where we still over-merge.
• The cluster is a product feature. Its size is the number of distinct sources, which the deck calls consensus. That number is a filter users rely on ("only stories 3+ sources are pushing"), it drives the coloured edge on the card, and it decides who was "first" in the leaderboards. Getting it wrong is visible.

Step 0: everything in English, first

Translation happens at ingestion, before clustering ever sees a push. Each notification's title, text and subtext are translated to English synchronously as it is stored (with a chain of fallback providers if the primary one fails), so by the time the clusterer runs, a Finnish push and a Portuguese push about the same event are two English strings. English-language sources are stored as-is. This is the single most important design decision: it turns a cross-lingual problem into a monolingual one and lets one cheap comparison serve every country.

The clusterer then keeps only the first 30 words of each push. Notification texts are short anyway; the cap stops the occasional 600-character "morning briefing" push from sharing a word with everything and hoovering up unrelated stories.

Step 1: cheap candidates, by lemma overlap

The live clusterer is a loop that runs continuously. Each pass it loads two sets of pushes: everything from the last 90 minutes, plus older pushes, up to six hours back, that already belong to a cluster with two or more sources. That way an established story keeps collecting late arrivals. Then, oldest first, it looks at each push that is not yet in a cluster and compares it with what came after it inside the 90-minute window.

The comparison is deliberately primitive. There are no embeddings in the real-time path. We run spaCy's small English model with everything switched off except the tokenizer and the rule-based lemmatizer. No parser, no tagger, no entity recognition, no vectors. What is left reduces each push to a set of lemmas: lower-cased, stop-words removed, single characters removed, and a hand-maintained list of "non-connecting" words removed. Two pushes are a candidate pair if their lemma sets share at least three items.

For a push being compared against an existing cluster, the cluster is represented not by one member but by its 17 most common lemmas across all members, a rolling profile of what the story is about. (The first six of those lemmas, if they fit in 42 characters, become the small tag chips you see on the card.)

Why so crude? Because it is fast, it is free, and it has a useful property: three shared content lemmas between two 30-word news pushes translated into English is a genuinely strong signal. Names, places and specific nouns survive lemmatisation; filler does not. Its weakness is that it knows nothing about meaning. "Trump falls" and "Trump's approval falls" share the same evidence. Which is why there is a step 2.

Step 2: the LLM referee

Every candidate pair from step 1 is put to a language model with one question. The system prompt is exactly: "Does these Push Notification belong together? I.e. are they about the same news story? Answer yes or no." The user message contains both pushes as Push(COUNTRY=…, SOURCE=…, TEXT=…). We use a small, fast model (currently gpt-4o-mini), ask for at most 20 tokens, and treat any answer that does not contain "no" as yes.

Two engineering details matter more than the prompt. First, the answer is cached per pair (an in-memory LRU of 20,000 pairs), because the same pair recurs on every pass while both pushes are inside the window. Second, and this is the part we are candid about, the call fails open: if the model API errors or times out, the pair is treated as "same story". We chose availability over precision: a provider outage degrades the deck to pure lemma-overlap clustering (more merges) rather than stopping clustering altogether (no consensus, no filters). It is the right trade for a live monitoring tool and the wrong one for an archive, which is one reason the archive gets a second pass (below).

Step 3: attach or create

If the referee says yes and the earlier push already has a cluster, the new push joins it. If the earlier push has no cluster, a new one is created and named after that push's id. So every cluster's name tells you which notification started it, and that is what the "who was first" leaderboards are built on. On every change we recompute two numbers from the database: n_size (pushes in the cluster) and n_sources (distinct source packages). The latter is consensus. The newest member is flagged latest so a consensus-filtered column can show one card per story, and the cluster gets a colour from a fixed palette by its running id, which is why neighbouring stories in the deck never share an edge colour.

One rule keeps the whole thing stable and also limits it: the live worker only ever adds. Once a push is in a cluster it is skipped as a candidate on later passes; the live path never splits or moves. Fixing mistakes is someone else's job.

The merger

The live clusterer's window is 90 minutes and it attaches greedily, so a big story frequently ends up as two or three competing clusters: one that started from a French push, one from a Swedish one, one from a robot headline. A separate service runs every five minutes over the clusters active in the last hour that have at least four sources, and considers every pair.

It reuses the same three-lemma pre-filter (computed over a sample of three pushes from each side, taking the best pair) so most pairs are rejected without a model call. Survivors go to the model with a stricter brief, "Be strict - only say YES if they are clearly about the same specific event or development", and a request for a one-word verdict plus a 50-word reason. Here the call fails closed: an API error means no merge. On YES, the newer cluster is folded into the older one, sizes are recomputed, colours and tags carried over, and a row is written to a merge audit table with the model's stated reason. Old cluster links keep working via a redirect.

The night shift: an agent re-clusters the day

Everything above is optimised for the next ten seconds. For the "who was first this week" leaderboards, correctness matters more than latency, and a completely different machine takes over at midnight.

The day's clusters are copied into a separate table and handed to a language-model agent with tools: show_clusters for a three-hour window, show_cluster, find_similar, move_push, merge_cluster, split_cluster_starting_at_push, orphan_push, create_new_cluster. This is where embeddings finally appear: find_similar embeds every push of the day with text-embedding-3-small and returns the 20 nearest neighbours above cosine 0.4 within ±60 minutes, but only as suggestions. The agent decides. The agent walks the day in overlapping windows (each three hours wide, stepping one hour), with a hard budget of about $30 per country-day, in two passes with a cheap model (gpt-5-mini) and a final pass where a stronger model (gpt-5.5) is shown only the diff between the raw clusters and the cheap model's result and asked to adjudicate.

The prompt is 250 lines long and most of it is worked examples of failure. Its core principle: "A good cluster contains notifications that all describe the same underlying development in an evolving news story, such that they could reasonably be summarized by one shared headline without feeling misleading or overly broad." And its most important rule, which the live clusterer structurally cannot follow: "A cluster should describe one newsworthy update, not one entire news cycle." Concretely: an outbreak's cause investigation, its case count, its spread and the government's reaction are four stories, not one; a Supreme Court ruling and the President's furious reaction are two; a live ticker's "match starts" and its "match won" are two; the same earthquake pushed in six countries is one.

If pushes answer fundamentally different questions, they are probably different stories. What happened? How dangerous is it? Who caused it? Where is it spreading? How are authorities reacting? From the re-clustering prompt

Today this corrector runs for Sweden, and its output feeds the Swedish toplists. It is the model for where the whole system is going: a corrector that runs every few minutes over the last two hours, with 48 hours of embeddings for lookups, that gradually replaces the merger and gives the live path a way to split.

What it gets wrong

These are the failure modes we know about. If you use consensus filters or the leaderboards, you should know them too.

1. Live coverage and sport. A live blog that pushes "follow along", "1-0", "red card", "full time" produces four pushes with heavily overlapping lemmas. The live clusterer merges them; the night agent's prompt has four separate examples about exactly this because it is the most common correction it makes.
2. Rich clusters get richer. A cluster's profile is its 17 most common lemmas. The bigger a story, the more generic that profile becomes, the more unrelated pushes can score three overlaps against it. Mega-events (a war, an election night) attract satellites. consensus_lt exists partly so you can filter those out.
3. The referee's outages become merges. Fail-open, as described. Rare, but when it happens the deck over-clusters until the API is back.
4. Same publisher, several apps. Consensus counts distinct packages. A group that pushes the same text through five regional apps looks like five sources unless we have mapped them; the Swiss hack above and a source-mapping table cover the cases we have caught. Regional syndicates in France and Belgium still inflate some clusters (visible in the year report's busiest-day table).
5. The language token. The candidate step appends the source language to the lemma set, so two pushes in the same original language get one overlap for free and effectively need only two shared content lemmas, while a cross-language pair needs three. It biases slightly toward same-country clusters. We are aware; it is on the list.
6. Append-only in real time. Once attached, a push stays attached until the merger or the night agent says otherwise; the deck can show a wrong membership for hours. The live corrector plan is the fix.
7. Translation quality is a ceiling. If a fallback translator garbles a push, the lemma step sees garbage. Everything downstream inherits it.

The numbers

Why a news desk should care

You do not need to know any of this to use the deck. But three product behaviours follow directly from it, and they are the ones people ask about:

• "3+ sources" means three distinct apps or robots, not three pushes. That is why it is such an effective noise filter, and why a publisher with many apps can occasionally inflate it.
• The first push in a cluster is "first". Leaderboards credit the source whose notification started the cluster (with wire-agency bylines excluded and article publish times allowed to beat notification times). If the live clusterer split a story in two, the night agent's version is what the Swedish toplists use.
• Clusters can be wrong for a while. If a card looks like it swallowed a neighbour, it probably did; open the cluster (the coloured edge) and you can see every member with its source and time. Corrections land in the merger and the night pass, and in the non-connecting-words list if it was one of those Tuesdays.

Questions, or a failure case we should see? hello@ao.news. We read all of them; some become prompt examples.