An active-learning loop for shelf annotation

Labeling is the recurring cost behind any shelf-recognition system: people marking what is in each photo so the model can learn. Cutting that bill only pays off if the labeling that remains is the labeling that matters. That is the job of our in-house labeling tool, Data Studio. It decides which shelves to label next and sends each task to the right person. It hands the person a short list of likely products, then feeds their corrections back into training. The result is a recognizer that keeps getting better without anyone labeling everything.

Routing a task to the right person

Who labels a shelf decides how fast and how accurately it gets done. Send the task to the right person and the same labeling budget buys more. So the tool routes each shelf to whoever is likely to be quickest and most accurate on it.

Every labeler has a daily limit and a set of product categories they know. The limit is their own weekly average plus a couple hundred items, so nobody is overloaded. A shelf goes to whoever labeled it most often before, by the majority of its last ten labelings. If that person is full, it falls back to others who know the same categories, least busy first. A shelf nobody has labeled before goes to a vision model, which reads the photo and picks the right group. People who go idle are dropped and their work is reassigned.

Shortlisting the likely products

Picking one product out of twenty thousand by hand is slow, and slow labeling runs the bill up. So the tool never makes a person search. Each task is a crop: one product cut out of a shelf photo. The crop comes with a short list of likely matches, and the person just confirms one.

The list is built from two things. First, the top guesses of a model that ranks likely products, a CatBoost ranker. Second, the products that model most often mixes those up with, learned from past corrections. With the best guess plus twenty of these mix-up pairs, the right product is in the list about 99.7% of the time. The list holds ten items or fewer in about 85% of cases.

Catching a new product early

Thinning the labeling has a cost, and it shows up as a blind spot. A product new to a shelf can wait days for its first label. It becomes reliably recognized only over about a month, as the system builds up reference photos of it. So we tried to flag new products automatically.

The most reliable signal is the recognizer’s own uncertainty. A photo it cannot confidently match is a candidate for something new. We tried seven versions, from per-shelf score cutoffs to comparing a shelf against its own recent photos. They catch a lot, but at a cost: the ones that caught the most were right only a few percent of the time. So this stays a helper that flags items for a person to check, never an automatic decision. It is the least solved part of the loop.

What the loop is worth

The whole point of a feedback loop is that the data it gathers makes the model better. It does. We took the photos that flowed back through Data Studio and retrained the product detector on them. We measured the result on a small, hard set of frames the loop itself had flagged.

Two things improved. The detector caught more of the products it should have, from about 92% to about 96%. That share is what we call recall. It also made fewer wrong calls: about 96% of its calls were correct, up from about 94%. That share is precision. In plain terms: fewer missed items on the shelf and fewer false alarms. One caveat: this is a controlled before-and-after on hard frames the loop picked. It is not a site-wide number. The direction is what counts.

What we improved next

The 2025 round of work was mostly about making each labeling task faster, which is the same as cheaper. Bigger shelf photos and bigger product crops. The likely-products list as a one-click action. A view of what a product already looks like in the gallery. And a button to flag a photo as bad or unclear, so a person can mark a hard case and move on.

Five things we took away

Spend labeling where it moves the model. The loop exists so the next label is the most useful one available.

Route by who knows the shelf. The person who labeled a fixture before is usually the fastest and most accurate on it again.

A short list beats a search box. Lists from the ranker plus its mix-up pairs put the right product in front of the person about 99% of the time.

New-product detection is hard. Uncertainty signals and shelf-to-shelf comparison catch new products, but they are right only a few percent of the time. Keep a person in that loop.

Feedback data pays back. Photos from the loop lifted the detector’s catch rate from about 92% to about 96% on hard frames.

The loop feeds the two models behind recognition. One fingerprints each product, the embedder. The other builds the likely-products list, the CatBoost ranker. It pairs with cutting the labeling bill and keeping the reference library fresh. If you are scoping a build like this, the shelf-monitoring use-case is where it becomes a project.