Keeping a million-vector search space fresh
The recognizer matches each product crop against a gallery of labeled crops, stored as vectors in Qdrant. We call it the search space. It grows to a few million vectors, and left alone it goes lumpy and stale. This note is about keeping it fresh. Why recent labelings beat clever cleanup. How we swap the whole index when the embedder changes. And why a smaller, fresher gallery is also more accurate. The recognizer itself is in part one.
What the search space is
Every labeled crop becomes a vector with a payload: its product, store, and category. A new crop is matched to its nearest neighbor, and the neighbor’s label is the prediction. The gallery for one chain runs to around 740,000 crops across roughly 6,000 products. It is wildly unbalanced. A popular cola has thousands of crops, a holiday candy box has one or two.
Freshness beats cleanup
A fixed camera sees the same shelf for months, and the shelf changes. Packaging refreshes, light shifts with the season, the camera drifts a pixel at a time. So a crop from last week matches an incoming frame better than a crop from six months ago. The simplest lever turned out to be the strongest one: keep only the recent labelings.
Concretely, we keep the chronologically last third of the gallery, plus the last ten crops of every product so nothing rare drops out. That cuts the index to about a third of its size. On one embedder it moved accuracy from around 96.9% to around 97.1%, slightly up, on an index three times smaller. We hold it there by rolling. When a new labeling round comes in, the oldest round goes out, and the size stays flat.
| Search space | Crops | acc@1 |
|---|---|---|
| Full | around 732,000 | around 96.9% |
| Last third, plus 10 per product | around 246,000 | around 97.1% |
| Last 20 photos per product | around 252,000 | around 97.1% |
One embedder, one chain. The metric is the labeler-confirm proxy used across the series, so read it as directional. On a larger production index the trade reads the same way. Halving the gallery from around 926,000 to around 534,000 crops cost about half a point overall. It gained on fresh products, the ones that matter most. The classes that drag the metric down are the stale ones. Products whose only examples are months old score about five points lower than products with a recent crop.
Why cleanup mostly failed
The obvious move is to clean the gallery: drop duplicates and bad crops. We tried, and it mostly failed. A cosine-similarity dedup that removes a fifth of the index costs around 0.4% accuracy. To stay under a tenth of a percent, you can remove at most about 5%. Perceptual hashing lost the small text and color that are our only signal between near-identical packs. Connected components with farthest-point sampling did best: around a third of the index removed for a 0.3% drop. Aggressive combined cleaning cost more than a full point. Outlier detectors like DBSCAN and Cleanlab threw out fine crops shot at odd angles. The one automatic filter that helped was a majority vote: drop a crop that never lands in the top of its neighbors’ votes. The fuller post-mortem is in experiments that did not ship.
One nuance is worth keeping. Cleaning the gallery hurt. Cleaning the training data helped. The same junk crops that were safe to keep in the index were worth removing before training the embedder.
Hot-swapping the whole index
A retrieval model has a catch a classifier does not. There is no class head, so when the embedder weights change, every vector in the gallery is now in the wrong space. The whole index has to be re-embedded before the new model can serve. We do it in the background. A new set of weights lands. We build a fresh Qdrant collection from the new model. When it is full, we swap the pointer from the old collection to the new one. New products keep arriving through an API in the meantime. They are embedded on the fly, so search finds them after a couple of labelings.
Five things we took away
Freshness is the cheapest accuracy you will find. Keeping the recent third of the gallery shrank it threefold and nudged accuracy up, because retail products drift in appearance.
Stale classes are the weak ones. A product whose only crops are months old scores well below one with a recent example, so age matters more than crop count.
Automatic cleanup fails on shelf crops. Dedup, perceptual hashing, and outlier detectors all cost more accuracy than they saved on non-studio retail images.
Clean the training set, keep the index fresh. The same crop can be worth dropping from training and worth keeping in the gallery.
Plan the hot swap from day one. With no class head, every weight update re-embeds the whole index, so the swap has to be a background, pointer-flip operation.
This gallery feeds the embedder and realgram and the CatBoost ranker. Keeping it fed cheaply is the other half of the job. That is cutting the annotation bill and the active-learning loop. If you are scoping a build like this, the shelf-monitoring use-case is where it becomes a project.