METRO Markets: Merging near-duplicate product listings into one

On an open marketplace, many sellers list identical goods, so one product naturally appears as several near-duplicate listings. The old approach compared the wording of titles, which breaks on word order, units, abbreviations, and the six languages the catalog runs in. Two listings for one product can read very differently.

The harder part is telling a real duplicate from a variant. A shoe comes in many sizes and a plate in many colors, and those are genuinely different items that should stay separate. Go by the words alone and you get it wrong both ways: the old method scored about 0.83 on shoes and about 0.50 on umbrellas, because short umbrella titles hide the one detail that sets the products apart.

Product categories are wildly different in size. Some hold a handful of items, others tens of thousands. And the effort to find matches grows much faster than the catalog itself, because every item has to be checked against the others in its category.

So a few giant categories swallow most of the work. In one survey, candles alone accounted for more than half of every comparison the system had to make, and a few categories made up the bulk of the rest. That is why a single one-size model cannot serve the whole catalog. We handle each category on its own and keep even the largest ones inside a tight compute budget.

What matters is whether the final product groups come out right, because those groups are the listings a shopper actually sees. A system can look almost perfect at single yes-or-no match decisions and still produce a messy catalog. So we judge it on the finished groupings and treat the per-decision score as only a side check.

One wrong match is expensive. When the system links two unrelated products, it chains them into a single merged listing and drags everything underneath together. A tiny share of bad links barely moves the per-decision score, yet it can wreck what the customer sees. We also watch the scoring itself: early on, a measurement quirk made one category look like a total failure when it was actually fine. So the headline is always the quality of the finished groups, measured with care.

We reached for the fashionable approaches first and got nowhere. Comparing product photos went wrong, because real variants often look alike and different sellers reuse look-alike studio shots. Reading the words for their meaning was no better.

What separates two products is usually a single detail: a size, a volume, a model code. Those approaches blur it away, so a 180 cm parasol and a 3.5 m one look identical to them. The method that worked compares two listings detail by detail and reads that number straight. It was the first to beat the old text-matching, and it became the core of the system. Getting there took months of dead ends.

The system works in plain steps, all on cheap hardware. First it narrows the field: it compares listings only within one brand and category, because variants never cross brands. That throws out almost all of the impossible matches for free. Then it compares each remaining pair, groups the matches into products, and runs a second pass that double-checks every group and splits any that were merged too eagerly.

It also picks the right method per category. Where there is enough labeled data, a specialist trained on that one category does the job and is the most accurate. Where there is not enough, one general-purpose model covers the category instead. A basic word-matcher sits underneath for categories no one has touched yet. The specialist is strongest where data is rich, and its lead grows on the biggest categories.

Built together with the METRO Markets team, the system is far more accurate than the word-matching it replaced. On an internal accuracy score from 0 to 1, matching rose from about 0.59 to about 0.88 across categories, with the biggest gains where the old method struggled most. Duplicates collapse into one product page, genuine variants stay as options, and it all runs on a single cheap machine that refreshes a category in seconds.

A few things are still open: a couple of newer ideas are still being tested or may be dropped, and the very largest categories have a known limit we have not lifted yet. The system is live in production at METRO Markets, which owns the live numbers.