Breaking a CV quality ceiling: yes/no labeling and an incremental data pipeline

Brickit is a mobile app that photographs a pile of scattered LEGO bricks, recognizes each one, and suggests what you can build from them. This is how its recognition stack broke through a quality ceiling that had held for about a year.

After a few years the project had data, working models, and users, and quality stopped growing. The only real driver left was more data on the rare classes, and direct labeling could not mine them. So we changed the labeling question. The old task asked a labeler to name the class of each object. The new task asks only whether a proposed pair of a box and a class is correct, a yes or a no. That turned labeling into a fast binary job, fed by the model’s own guesses and steered toward the rarest and most error-prone classes. To run that loop in production we built Datapipe, an incremental and reactive data framework. Together they restarted quality at around 0.5-1% absolute per month, sustained for quarters. On the long arc, the weighted F1 of the classifier moved from around 0.44 to around 0.76. What the approach still cannot do is separate near-identical classes, which needs a separate curation pass.

Recap

The input is a photo with 500-1500 small objects. The output is a class for each object, drawn from about 1500 of the most common LEGO parts, out of around 30,000 that exist. The stack runs in two stages.

The detector finds everything that looks like a brick and draws a bounding box, with no attempt to classify. We use a YOLO-family model. On one demo frame it found 498 boxes: 467 true positives, 31 false positives, and 39 false negatives.

The classifier then takes each box and picks one of the roughly 1500 classes plus a few service labels. It is a ResNet-like model, one pass per box.

We split the stages on purpose. End-to-end detection and classification produced many opaque errors that are hard to interpret. Separate stages make debugging easier, give more control, and cut the count of strange failures.

The scale is large. The database is over 1.05 TB, the heaviest table over 210 GB, object storage over 2.7 TB. There are about 124 tables (62 train and test pairs), 59 pipeline steps, 194 retrained classification models, and 4 detection models. Along the way: 108 frozen datasets, 86 schema migrations, over 2.5 TB of Docker images in the registry, and more than 40 Metabase dashboards.

The quality ceiling

After a few years the project had data, working models, and users. Quality stopped growing. The one lever left was more data, and specifically on the rare classes. Rare classes show up in under 0.1% of cases, and direct labeling barely surfaces them.

Why direct labeling stops scaling

Labeling here is brutal. One frame holds about 1500 boxes. Direct labeling means picking a class out of 1500 for every box on the frame. That is around one week of human work per frame. It is expensive, and it is uncontrollable. You cannot ask in advance for more data on rare classes. You pay first and find out what you got afterward.

Reframing labeling as yes/no checks

The fix was to change the question. The old question asks which class an object is. The new question asks whether a proposed pair is correct. A hypothesis is a box and a class together, and the labeler answers yes or no. This is active learning in production, with no magic in it. Even a weak classifier is a useful signal. If the model says there is a 10% chance a box is some rare class, that is tens of times better than a random guess.

The loop is short. Take real user photos. Generate hypotheses, each a box and a class. Group them by class. Prioritize the rarest and most error-prone. Send those for binary labeling.

Tinder: the labeling tool

We built a separate tool, internally called Tinder. The labeler sees a box and a render of the proposed class, then swipes: yes or no. It works because it is cognitively simple, it fits a phone, and it runs at tens of thousands of decisions a day. The target conversion is 1-5% positive answers. Negatives cost almost nothing. Positives are gold.

Why off-the-shelf pipelines failed

The loop only pays off if one confirmed label flows through the system fast and stays correct. Standard tools did not fit. Batch schedulers like Airflow and Prefect react poorly to small point changes. Streaming systems trade correctness for real time, and they handle expensive operations like model training and human labeling badly. The shared pain is incrementality. It is either missing or hand-rolled at every step, and it is easy to end up with inconsistent data.

So we built Datapipe. It runs in a mixed mode, batch and incremental. It tracks changes at the row level, knows the dependencies between tables, and recomputes only what is needed. The mental model is a CAD system, with data in place of geometry. A change at the start of the pipeline propagates downstream on its own.

The payoff is reactive labeling. A labeler confirms a hypothesis, the box gets its final class, and every other hypothesis for that box drops out of the queue at once. The update travels the full pipeline, around 59 steps and dozens of tables over hundreds of gigabytes, in seconds. It runs in production on Kubernetes.

Metrics without self-deception

Two metric problems shaped the work. First, classical accuracy and F1 correlate poorly with what the user feels. We report weighted accuracy, with each class weighted near its real prevalence, plus separate precision for problem subsets of classes. We also tried to compute a “user suffering” metric. It did not work out.

Second, the usual trap is that the model and the test set change at the same time, which makes scores incomparable. We keep the test set live. When it changes, we recompute the metrics of old models retroactively. Old models then drift down over time, because we are measuring reality more honestly. There are fewer illusions of progress.

What did not work, and why

Synthetic data did not carry the project. The bootstrap used Blender renders of LEGO plus hand labeling. A classifier trained on renders transferred poorly to real photos, and real data stayed necessary.

The “user suffering” metric did not survive contact with the data.

Binary labeling has a real blind spot. It separates very similar classes poorly, and it can introduce systematic errors. Different variants of one part can look identical at a glance, so a yes or no on a single hypothesis does not always pin the right one. The fix is a separate dataset-curation pass: view every box assigned to one class, catch the mistakes quickly, and clean the set. The cleaning itself lifts quality.

What this adds up to

Five things carry over to any similar build. A quality ceiling is almost always a data ceiling. Reframing the task can matter more than a new model. The gain here came from changing the labeling question, and the architecture stayed the same. Active learning is a process architecture you design, and no library gives it to you for free. Data pipelines in ML need three properties at once: incrementality, correctness, and reactivity. And sometimes you have to build your own framework.

The pattern fits any task shaped as “find the objects and classify them against a fixed, known set of classes”: LEGO parts, screws and fasteners, industrial components, product catalogs. We ran a similar yes/no loop on retail shelves in the shelf annotation loop. If you are scoping a build like this, the mlops-platform use-case is where it turns into a project.