Datapipe: recompute only what changed in a 59-step ML pipeline

This is a companion to breaking a CV quality ceiling, which covers how we reframed labeling for Brickit. That post is about the labeling loop. This one is about the data framework underneath it, Datapipe.

Brickit’s recognition pipeline runs 59 steps over about 124 tables and more than 1.05 TB. Off-the-shelf orchestration did not fit. Batch schedulers recompute too much and react slowly to small changes. Streaming systems give up correctness for low latency, which is the wrong trade when a single step costs a model training run or a human decision. So we built Datapipe. It tracks changes at the row level, knows the dependencies between tables, and recomputes only what a change touches. One confirmed label now flows through the whole pipeline in seconds, in production on Kubernetes. We also keep the test set live and re-score old models against it, which removes most illusions of progress. The cost is a real maintenance surface: 86 schema migrations and over 2.5 TB of images so far.

What the pipeline feeds

The framework exists to serve one job: keep a continuously growing training set consistent. Brickit recognizes about 1500 LEGO parts in a photo, in two stages. A detector draws a bounding box around each part. A classifier then names each box. An active learning loop turns real user photos into hypotheses, each a box and a class, and sends them for yes or no labeling. The labeling loop has the full story.

The data keeps moving. The train set grew from 111,300 to 360,600 rows, the test set from 25,800 to 77,900. We retrained 194 classification models and 4 detection models, froze 108 datasets, and watch it all on more than 40 dashboards. Every confirmed label, every new photo, every dataset freeze is a change the pipeline has to carry through to the metrics without corrupting anything.

Why off-the-shelf pipelines fell short

Batch schedulers like Airflow and Prefect are built for whole-stage reruns. A small change, one confirmed label, does not map cleanly to that model. You either rerun far more than you need, or you hand-roll partial reruns step by step.

Streaming systems go the other way. They trade correctness for low latency. That is fine for cheap, append-only events. It is wrong here, because the expensive operations in this pipeline are model training and human labeling, and you cannot afford to run them on data that might be stale or inconsistent.

The shared pain is incrementality. It is either missing or hand-rolled at every step. Hand-rolled incrementality is where inconsistent data comes from: a step that forgets one upstream change leaves a stale row, and a stale row in a training set is a silent quality leak. We learned that the slow way, by hand-rolling it first.

What incrementality has to mean here

The requirement has four parts at once. Track each change at the row level. Know the dependencies between tables. Recompute only the rows downstream of a change. And keep correctness, so no stale row survives.

The constraint that forces all four is cost. A recompute can mean a training run or a human swipe, so recomputing everything on every change is off the table. Dropping correctness is also off the table, because a corrupted training set costs more than it saves. You need the small, exact recompute, every time.

Datapipe, a CAD for data

Datapipe runs in a mixed mode, batch and incremental. It tracks changes at the row level, holds the dependency graph between tables, and recomputes only what is needed. The mental model is a CAD system, with data in place of geometry. A change at the start of the pipeline propagates downstream on its own, and only along the paths that depend on it.

Reactive labeling in real time

The payoff shows up at the labeler’s screen. A labeler confirms a hypothesis. The box gets its final class. Every other hypothesis for that box drops out of the queue at once, because they are now impossible. The update travels the full pipeline, around 59 steps and dozens of tables over hundreds of gigabytes, in seconds. None of that is a nightly rerun. It is the direct, immediate consequence of one row changing.

Metrics without self-deception

The usual trap is that the model and the test set change at the same time, which makes two scores incomparable. We keep the test set live. When it changes, we re-score the old models against it retroactively. The old models then drift down over time, because we are measuring reality more honestly than before. Watching your own old models quietly get worse builds character. There are fewer illusions of progress, which is the point.

What it costs

A custom framework is a real surface to carry. So far it is 86 schema migrations, over 2.5 TB of Docker images in the registry, about 124 tables, and a heaviest table over 210 GB. The 86 migrations are one number we keep off the recruiting page. Row-level incrementality also asks for schema discipline, since every change has to be tracked and every dependency declared. And retroactive scoring can look like a regression to someone reading a dashboard, so you have to explain why old numbers are allowed to fall. None of this is free. It paid off here because the pipeline is long, the steps are expensive, and the data never stops moving.

What this adds up to

When expensive human or model steps sit inside a data loop, batch and streaming both miss. Batch recomputes too much and reacts too slowly. Streaming gives up the correctness you cannot give up. What the work needs is three properties together: incrementality, correctness, and reactivity. When no tool gives you all three, building your own can be the cheaper path over a multi-year project.

The pattern reaches past object recognition. It fits any continuously labeled ML dataset: anywhere new data lands all the time and a confirmed label should reach the metrics without a full rerun. If you are scoping a build like this, the mlops-platform use-case is where it turns into a project.