Cutting the annotation bill without losing accuracy
Recognition lives on human labeling, and labeling never stops. Every shelf gets re-labeled again and again as products move, so the bill is recurring and it dominates the cost of the system. We were labeling around two million crops a month and wanted to reach about 375,000. This note is how we cut most of that without losing accuracy, and why the cut came in smaller than the math promised.
Where the work goes
One store holds well over ten thousand unique products, and each gets re-labeled roughly ten times a month. Most of that is waste. A shelf that never changes does not need a fresh labeling every two days. The trouble is that the value of any single labeling is hard to see in advance, so the safe default was to label everything, often.
Tier shelves by how much they change
The fix is to label a shelf as often as it actually changes. We score each shelf by its edition rate: how much its contents turn over. Then we sort shelves into tiers and give each tier a cadence. Stable shelves, the ones under a 5% edition rate, get labeled far less often. Medium shelves less often than the busy ones. The busiest shelves, over 25%, keep their frequency. When we drop a labeling, the shelf state is rebuilt by carrying each product forward from its last good labeling, so the record stays complete.
| Shelf tier | Edition rate | Cadence |
|---|---|---|
| Simple | under 5% | least often |
| Medium | 5-25% | less often |
| Complex | over 25% | unchanged |
The shipped scheme. Stable shelves are re-labeled far less often than busy ones.
The math said 2x, production said 1.66x
In the lab the scheme should have halved the labeling. Production delivered a 1.66 times cut. The gap was a people problem. The model had nothing to do with it. A task is created when the last task on that camera is more than 22 hours old. Doubling that window should halve the tasks created. But the labelers already had a backlog. With around 200 cameras and a throughput near 150 a day, the old scheme cleared 300 over two days. The new one cleared 200: 150 on the first day and 50 the next. That is the 1.66. It is a queue effect, plain and simple.
Did it hurt accuracy
We set a hard bar. More than a 5% drop in recognition accuracy is unacceptable. The shipped scheme stayed well inside it. Overall accuracy moved from around 87.6% to around 86.3%, a little over a point. The search space shed well over a hundred thousand crops. Fresh products held up, which is where the risk was. So most of the labeling went away for around a point of accuracy, a trade we will take. As across the series, that accuracy is the share of predictions a labeler confirms, measured on one chain, so read it as directional.
Four things we took away
Label a shelf as often as it changes. A stable shelf re-labeled every few days is paying for information that did not change.
Backfill the gaps. Carrying each product forward from its last good labeling keeps the shelf record complete between labelings.
Lab savings are an upper bound. A labeling queue meant a planned 2x cut arrived as 1.66x, because the bottleneck was annotator throughput.
Set an accuracy floor first. A 5% drop was the line, the cut cost about a point, so it shipped.
Cheaper labeling pairs with a fresher index: keeping the search space fresh. The tool that decides what to label, and routes it, is the active-learning loop. The recognizer it all feeds is the pipeline overview. If you are scoping a build like this, the shelf-monitoring use-case is where it becomes a project.