Editors see storylines, because the system clusters every item before it hits the desk
DBSCAN and BERT-based clustering that groups news items by storyline, generates a short description per cluster, and ranks by category. Built for editorial teams and media-monitoring operations.
Discuss a pilot →Quick facts
- Business size
- Media operations, news monitoring services, editorial teams
- Timeline
- 4-6 weeks test, 2-3 months pilot, 3-5 months production
- Budget range
- Pilot from €25k.
- Hardware
- Cloud-based. CPU for inference; GPU helpful for embedder training.
- Data needed
- Sample news corpus. Category taxonomy.
- Evolution
-
- Genesis
- Custom-built
- Product
- Commodity
A vendor sells this result ready-made. We set it up and tune it to you.
What this scale means
Further right means more proven and cheaper. Further left means newer and riskier. Here is the test for each step.
- Commodity
- You could get the result yourself from a ready service, with almost no work. We rarely take these on.
- Product
- A vendor already sells this result turnkey, like shelf recognition from Trax or document reading from ABBYY. If one of them fits you, use it. You come to us when it does not: when it has to run on your own servers, cost less, or fit systems the product cannot reach.
- Custom-built
- Vendors sell only the parts. A tool like Tableau hands you charts, but the dashboards and metrics for your business still have to be built. That build is the work, and that is us.
- Genesis
- The approach exists but does not work reliably yet. You are betting on it maturing, so it costs more and carries more risk.
Expected outcomes
The Problem
Editorial and media-monitoring teams drown in volume. Hundreds of related items arrive about one event from different sources. The editor has to sort the duplicates, identify the angle, and decide what to publish. Most of that time goes to sifting. Little of it is editing.
Generic news aggregators (Google News, Yandex News) cluster at the title level, which works for obvious cases and breaks down on nuance. “X said Y about Z” and “Z denies X’s claim about Y” should sometimes cluster as one story, and sometimes stay apart as separate developments.
Dedicated NLP for news clustering, DBSCAN-style density-based clustering on top of semantic embeddings, handles this much better. Items that share semantic content cluster; outliers stay separate; cluster sizes stay manageable for editor processing.
What the Solution Does
A news-processing pipeline that does the sifting before editors see the feed.
- Ingest, news items from your sources (RSS, APIs, scrapers).
- Embed, BERT-based embedder converts each item into a vector.
- Cluster, DBSCAN groups items into storyline clusters, with parameters tuned for cluster homogeneity and manageable size.
- Summarize, short description generated per cluster.
- Classify and rank, clusters tagged by category, ranked by importance for the target user.
- Editor surface, dashboard with clusters, summaries, classifications, items within each cluster.
Where It Fits
This makes sense if you…
- Operate editorial or media-monitoring with substantial news volume
- See real cost from editor time spent sifting
- Want consistent clustering across editors
- Have a category taxonomy, or are willing to define one
This is probably not the right time if you…
- Process small news volume
- Need real-time alerting on individual items (this is batch-clustering; alerting needs different architecture)
Business Value
Editor time recovery. Editors typically spend 40-70% less time sifting duplicate and related items, depending on the corpus. They focus on the editorial decisions.
Consistency. One clustering logic applies to every item. Inconsistencies between editors drop.
Coverage. Auto-generated summaries mean the editor sees every storyline, even the ones they had no time to read.
Adjacency to legal and compliance. The architecture also applies to legal precedent clustering (group cases by legal storyline), patent monitoring, and regulatory tracking.
How It Works
1. Embedding
BERT-based embeddings (multilingual where needed) per news item. Spacy for preprocessing (entity extraction, tokenization).
2. DBSCAN clustering
Density-based clustering with parameters tuned for:
- Cluster homogeneity (items in a cluster share one storyline)
- Cluster size (manageable for an editor, typically 3-15 items)
- Coverage (most items map to some cluster, with very few outliers)
The DBSCAN parameters (eps, min_samples) get tuned per corpus. We’ve found settings that produce 85-95% homogeneous clusters on our published deployments, depending on the source mix.
3. Storyline summarization
For each cluster, an LLM or extractive summarizer produces a 1-2 sentence storyline description.
4. Classification and ranking
Each cluster classified against your category taxonomy. Clusters ranked by importance signals (volume, source quality, time-decay).
5. Editor surface
Dashboard / API for editors. Drill-down from cluster to items. Edit / publish actions feed back into the system.
Stack
Spacy handles preprocessing. BERT-based embeddings (multilingual where needed) feed DBSCAN clustering, and an LLM writes the summaries. Datapipe runs the pipeline, with Tensorflow for any custom classifier training.
What You Need to Make This Work
Data. Sample news corpus, category taxonomy.
Integrations. News source feeds. Editor workspace / CMS integration.
Hardware. Cloud-based.
Team. Editorial lead. Data engineer.
Implementation Roadmap
1. Test (4-6 weeks)
Embed sample corpus. Tune DBSCAN parameters. Validate cluster quality against editor judgment. Output: working clustering with measured cluster quality.
2. Pilot (2-3 months)
Production deployment for one news domain. Wire up editor surface. Tune summarization. Output: working deployment with editor adoption signals.
3. Production (3-5 months)
Full rollout, additional domains, continuous improvement.
Keep in Mind
- DBSCAN parameters matter. Wrong settings produce too-broad or too-narrow clusters. Tuning is real work.
- Storyline summarization is LLM-grade. Imperfect; editors review for high-stakes content.
- New event types may not cluster cleanly. Novel stories sit as singletons until similar items arrive.
- Multi-language requires tuning. Embeddings handle it; clustering parameters may differ.
FAQ
Why DBSCAN and not k-means?
K-means requires choosing the number of clusters in advance, impossible for news (storyline count varies daily). DBSCAN auto-detects clusters from density, no advance choice needed.
Can this work for non-news (legal cases, patents, etc.)?
Yes. The architecture carries over, with different embedder fine-tuning per domain.
How fresh is the clustering?
Configurable, typically hourly batch, but near-real-time is possible.
Ready to Discuss?
If you operate editorial or media monitoring at meaningful volume, this is a focused pilot. We’ll discuss your sources, your editor workflow, and tell you what to expect.