Comparing twenty embedding models for retail recognition
Before fine-tuning your own embedder, it is worth asking whether a big foundation model already does the job. We asked it properly. We took more than twenty image embedders, ran them on the same set of real shelf crops, and ranked them by top-1 retrieval accuracy. The fine-tuned ViT-B/32 we already had in production won, at around 95% against the best foundation model’s 89%.
This is a companion to building the embedder, which is about how that small model was trained.
The benchmark
Every model saw the same test: about 282,000 product crops pulled straight from store cameras, with no label cleaning. For each crop we take its embedding, find the nearest neighbor in a labeled gallery, and read off the product. The score is top-1 accuracy. One shortcut to call out: the full metric takes over an hour to run. During iteration we used a 500-product subset, which reads a little high. The numbers below are the full set.
The leaderboard
| Model | Backbone | acc@1 |
|---|---|---|
| Our embedder, fine-tuned | ViT-B/32 | around 95.4% |
| Our embedder, RP2K only | ViT-B/32 | around 92.9% |
| Amazon Titan Multimodal G1 | undisclosed | around 89.4% |
| InternViT | InternViT-6B | around 88.9% |
| Our embedder, LAION only | ViT-B/32 | around 88.5% |
| M2-large | M2 | around 87.4% |
| ImageBind | ViT-H/16 | around 86.3% |
| DINOv2 | ViT-B/14 | around 83.8% |
| EVA-CLIP | ViT-B/16 | around 79.8% |
Top-1 retrieval accuracy on about 282,000 uncleaned shelf crops. One task, one catalog. The ranking is directional, a signal for this benchmark only.
Bigger was not better
The headline is the gap between the top row and the rest. InternViT carries 6 billion parameters. Our embedder is a ViT-B/32, orders of magnitude smaller, and it scored about 6 points higher. Amazon Titan, the strongest model we did not train, still landed around 89%. On this task, domain fit beat scale every time.
Preprocessing was part of the model
How a crop is resized changed the result by more than we expected. Our embedder resizes the long side to 224 pixels and pastes the crop onto a centered canvas, so nothing is cut. Models that center-crop to a square lose the edges of the package. EVA-CLIP dropped around 2 points under a center crop against a plain resize. Titan scored the same at 224 and 336 pixels, so a bigger input bought nothing.
Some models would not even fit
A few well-known models could not do the job at all. SAM, built for segmentation, would not run on crops this small. BLIPv2 returned no single clean embedding, only a token tensor and a projection of different sizes, so we had to pool it by hand. Amazon Titan works through an API with an undisclosed architecture, which makes it a black box you cannot fine-tune. We also tried concatenating our vector with Titan’s. It added nothing.
Text matching was weaker still
Earlier we tried the other direction: match a product by its name. We embedded each product name and each image with a CLIP model and searched text to image. On a catalog subset the best configuration reached around 42% top-1. Reading a shelf by matching names to pictures is far weaker than matching picture to picture.
Why the small one wins
The pattern is simple. A model fine-tuned on in-domain retail crops beats a general model that has never seen a price tag. We pretrain on RP2K, then fine-tune on store data. Scale does not close that gap: a 6-billion-parameter backbone still trails a fine-tuned ViT-B/32. And the preprocessing that keeps the whole package in frame is part of why.
Four things we took away
Try the foundation models, then fine-tune anyway. The best off-the-shelf embedder still trailed our small fine-tuned one by about 6 points.
Domain fit beats parameters. A ViT-B/32 beat a 6-billion-parameter model on this task.
Preprocessing is part of the model. A center crop that clips package edges cost around 2 points.
Black-box API embeddings do not fuse for free. Concatenating a paid external vector with ours added nothing.
How we trained the winner is in building the embedder. If you are scoping a recognizer, the shelf-monitoring use-case is where this becomes a project.