Supplier feeds in, marketplace listings out, without armies of moderators

The Problem

Marketplaces don’t make their own products. They ingest them from suppliers, manufacturers, wholesalers, distributors, and republish them in a unified catalogue. The supplier feed and the marketplace listing rarely use matching vocabulary. Categories differ. Attributes are named differently or missing. Brands get listed under the manufacturer’s preferred spelling and the marketplace’s preferred spelling. One physical product comes through three suppliers under three different names.

Manual workflows exist: moderators handle each new supplier feed, mapping categories, normalizing attributes, deduplicating against the existing master. At marketplace scale (hundreds of thousands of SKUs, multiple new suppliers per week) the moderator team becomes the bottleneck. Onboarding a new supplier takes days. Catalogue quality drifts as moderators rush.

The problem shows up in adjacent contexts. Pharma marketplaces match SKUs across suppliers (the pharma e-commerce platform case). B2B distributors normalize manufacturer catalogues. Multi-marketplace sellers republish one product across a major CIS marketplace, Wildberries and Yandex.Market, each with its own taxonomy.

What the Solution Does

A pipeline that takes a supplier feed and produces a marketplace-ready listing for each item, with categorization, attributes, deduplication against your existing master, and normalization to the target marketplace taxonomy.

1. Ingest, read supplier feed (CSV, API, XML, whatever the supplier provides).
2. Embed, convert each product name and description into a semantic vector (BERT-based embedder).
3. Categorize, KNN against your target marketplace taxonomy (3,500 or more category trees handled in production).
4. Extract attributes, NER pulls brand, weight, type, size, packaging from free-text names.
5. Match against master, fuzzy match to existing SKUs to detect duplicates and route them to the right master record.
6. Normalize for target marketplace, map brand names, attribute values, and units to the target marketplace’s vocabulary.
7. QC, items with low confidence get routed to moderators; their decisions feed retraining.

Where It Fits

This makes sense if you…

• Operate a marketplace (general or vertical) ingesting feeds from external suppliers
• Sell on multiple marketplaces and republish the same products with different taxonomies
• Are a B2B distributor normalizing manufacturer catalogues
• See real moderator-team cost or supplier-onboarding latency as a business constraint
• Have a target taxonomy that is documented (or are willing to document it during pilot)

This is probably not the right time if you…

• Run a curated marketplace where every SKU is hand-onboarded for editorial reasons
• Have a target taxonomy that’s not consistently defined, you can’t automate to a moving target
• Process so few new SKUs that moderator overhead is trivial
• Need full visual verification of products before listing (then visual search needs to be part of the workflow)

Business Value

Throughput. A trained pipeline categorizes and attributes thousands of items per second per CPU core. Supplier onboarding goes from days of moderator work to hours of automated processing plus human-review of exceptions.

Catalogue consistency. One reference catalogue and one model classify every item, every time. The “moderator A categorizes differently from moderator B” problem disappears.

Cross-marketplace publishing. One engine handles re-categorization for different target marketplaces. List a product once internally, then publish it to a major CIS marketplace’s taxonomy, Wildberries’s taxonomy, Yandex.Market’s taxonomy, each with their own categories, attributes, and brand vocabulary normalization.

Pharma and regulated-vertical applications. Matching pharmaceutical SKUs across supplier feeds is a hard, regulated problem. We built the matching engine for a pharma e-commerce platform, doing pharma-vertical SKU matching across multiple supplier sources. The architecture applies to any regulated vertical with strong category-tree requirements.

How It Works

The architecture matches our product categorization engine, with an added normalization-to-target-marketplace layer.

1. Semantic embedding

BERT-based embedder converts product names and descriptions into 500-700-dimension vectors. The embedder is pre-trained on years of supplier feeds, fiscal receipts, marketplace catalogues, and retail master data, so it handles typos, abbreviations, and unusual formatting from day one.

2. Categorization

KNN against your target taxonomy’s reference catalogue. Confidence scores route low-confidence items to moderator review. Editable reference catalogue means category-level fixes happen without retraining.

3. Attribute extraction (NER)

One embedder, different head: for each word in the product name, the model predicts which named entity it belongs to (brand, weight, type, etc.). Extracted attributes get normalized.

4. Duplicate detection

The embedder again: the new item’s vector gets compared against your existing master via similarity search. Above a threshold, the item is a likely duplicate and routes to the merge workflow. Below, it is a likely new item and routes to listing creation.

5. Target-marketplace normalization

Map the categorized and attributed item to the target marketplace’s vocabulary. Brand names that differ between your master and the target marketplace get normalized (your “L’Oreal Paris” matches the target’s “L’Oréal Paris”). Packaging units get standardized.

6. Moderator QC and retraining

Low-confidence items go to moderators. Their decisions feed Datapipe-orchestrated retraining cycles. The reference catalogue grows over time; accuracy improves.

Stack

BERT-based encoder, KNN with vector indices, NER head on the same encoder, Datapipe for the data pipeline, custom QC layer for moderator workflows, integration via REST APIs with marketplace publishing endpoints.

What You Need to Make This Work

Data. Target taxonomy (your marketplace’s category tree). Historical mappings of input to correct listing if you have them: 5-10k examples is enough to start. Without historical data, plan for a 2-4 week labelling sprint at the start.

Integrations. Read access to supplier feeds (API, file drop, email). Write access to your marketplace listing system. Optional: integration with target marketplace publishing APIs (a major CIS marketplace, Wildberries, etc.).

Hardware. Cloud or on-prem. CPU sufficient for inference. GPU helpful for retraining (we can run on our side).

Team. A catalogue-ops lead who owns the reference catalogue. A moderator pool (yours or ours) for QC. A data engineer for supplier-feed integration (around 25-40 hours during pilot).

Implementation Roadmap

1. Test (4-6 weeks)

Pick one supplier feed and one target marketplace. Map historical data, measure baseline accuracy on categorization, attributes and duplicates. Output: written report with per-category accuracy, recommendations for pilot scope.

2. Pilot (2-3 months)

Production deployment for that supplier feed. Wire up moderator QC. Build dashboards for catalogue ops. Output: working deployment, measured supplier-onboarding speed improvement, go/no-go on multi-supplier rollout.

3. Production (4-6 months)

Add additional suppliers, additional target marketplaces. Continuous retraining via Datapipe. Quarterly accuracy review. Your team owns the reference catalogue and moderator workflow; we stay on for embedder retraining and edge cases.

Keep in Mind

Known limits:

• Target-taxonomy quality is the ceiling. A marketplace taxonomy with overlapping or poorly-defined categories will produce a model that struggles exactly where humans do. Sometimes taxonomy cleanup is the most valuable fix.
• Duplicate detection is probabilistic. One product through different suppliers can be detected with high precision when supplier-side data is rich. It is harder when names are vague or attributes missing. We surface the confidence so you can see it.
• Pharma and regulated verticals are real work. The pharma e-commerce platform case had a well-defined target catalogue, which is why accuracy was high. Less-curated regulated verticals need taxonomy work first.
• Cross-marketplace mapping requires per-marketplace data. Mapping to a CIS marketplace’s taxonomy requires examples of items correctly classified in that marketplace. Each new target marketplace is its own data collection effort.
• Moderator QC is not optional. Even at 95% accuracy, 5% of items are wrong. Auto-publishing all of them creates catalogue rot. The QC layer is how you go from 95% model accuracy to 99% or higher catalogue accuracy.

FAQ

Can this handle pharma-specific requirements (INN matching, dosage equivalence, etc.)?

Yes. The pharma e-commerce platform case is the proof. The pharma vertical requires additional rule layers (INN-to-brand normalization, dosage parsing), and the underlying engine handles them via the NER and normalization layers.

How do we onboard a new target marketplace (e.g. Wildberries after a major CIS marketplace)?

Each target marketplace is roughly a 4-8 week effort: ingest the taxonomy, build the reference catalogue, validate accuracy. The engine doesn’t change.

What if suppliers send data in totally different formats?

The pipeline normalizes input. Supplier A sending CSV, supplier B sending API, supplier C sending email-attached XML, all get adapter layers. Sometimes the supplier-side data quality is the rate limit: weak input gives you a weak classification, no matter how good the model is.

Can we integrate with marketplace publishing APIs?

Yes. We’ve worked with a major CIS marketplace, Wildberries and Yandex.Market APIs. Each has its own quirks, and we adapt.

Does this work for B2B distributors as well as consumer marketplaces?

Yes. B2B catalogues have different patterns (more technical specs, less brand emphasis), and the architecture stays the same. We fine-tune the embedder for B2B vocabulary.

Ready to Discuss?

If you operate a marketplace or distribute via multiple marketplaces and supplier-onboarding latency or catalogue consistency is a real cost line, this is a measurable-ROI deployment. We’ll walk through your taxonomy, your supplier mix, and your target marketplaces, and tell you what to expect from a pilot.

Discuss your project