Stop showing customers ten copies of the same listing
Detect duplicate listings across suppliers, re-postings, and slight-rewording attempts. Built on the BERT-based embedder engine that classifies hundreds of millions of receipt lines for a national tax authority and a major CIS marketplace.
Discuss a pilot →Quick facts
- Business size
- Marketplaces, classifieds platforms, real-estate platforms with significant listing volume
- Timeline
- 4-6 weeks test, 2-3 months pilot, 3-6 months production
- Budget range
- Pilot from €30k.
- Hardware
- Cloud or on-prem. CPU sufficient for inference.
- Data needed
- Sample listings + historical duplicate decisions if available.
- Evolution
-
- Genesis
- Custom-built
- Product
- Commodity
No product gives you this. We assemble and train it around you.
What this scale means
Further right means more proven and cheaper. Further left means newer and riskier. Here is the test for each step.
- Commodity
- You could get the result yourself from a ready service, with almost no work. We rarely take these on.
- Product
- A vendor already sells this result turnkey, like shelf recognition from Trax or document reading from ABBYY. If one of them fits you, use it. You come to us when it does not: when it has to run on your own servers, cost less, or fit systems the product cannot reach.
- Custom-built
- Vendors sell only the parts. A tool like Tableau hands you charts, but the dashboards and metrics for your business still have to be built. That build is the work, and that is us.
- Genesis
- The approach exists but does not work reliably yet. You are betting on it maturing, so it costs more and carries more risk.
Expected outcomes
The Problem
Marketplaces and classifieds platforms accumulate duplicates the way social media accumulates spam. Sellers re-post their old listings for fresh prominence. Three different suppliers list the same product. A real-estate agent and the owner both post the same apartment. Users see the same listing five times, get frustrated, leave.
Manual deduplication doesn’t scale. Most platforms rely on user reports, which catch only the most obvious cases, and only after users complain. Algorithmic deduplication via plain string matching fails on minor rewording. Effective deduplication requires semantic understanding: “white Audi A4 2019” and “Audi A4 2019, white, excellent condition” are the same listing despite different text.
What the Solution Does
A deduplication engine that detects duplicate listings via semantic similarity and structured attribute matching.
- Embed, each listing converted to a vector using a BERT-based embedder (same as our product-categorization work).
- Find candidates, similarity search against existing listings returns top-N similar.
- Score, for each candidate, semantic similarity and attribute matching together produce a duplicate-confidence score.
- Decide, high confidence: auto-flag duplicate (merge / hide one). Medium confidence: route to moderator. Low confidence: keep both as distinct.
- Continuous improvement, moderator decisions feed retraining; new evasion patterns get caught faster.
Where It Fits
This makes sense if you…
- Operate a marketplace / classifieds / real-estate platform with > 10k listings
- See real cost from duplicate complaints, user churn, or content quality
- Have content-rich listings (descriptions, attributes, images)
- Want to deduplicate at upload time, ahead of any user report
This is probably not the right time if you…
- Run a small platform where moderation by hand suffices
- Have listings so unique (one-of-a-kind items) that duplication is rare
- Can’t enforce deduplication policy, algorithmic detection is useless without action authority
Business Value
User experience. Customers stop seeing the same listing repeatedly. Browse / search results improve.
Operator effort reduction. A 60-85% reduction in user-reported duplicate complaints, depending on baseline. Moderators handle the edge cases while the system clears the obvious ones.
Content quality signal. Duplicate analytics surface patterns: which sellers re-post most, which categories have most duplication, which formats get evaded. Strategic decisions follow.
Real-estate specific value. Multiple agents listing the same property is a common quality problem in real-estate platforms. Deduplication consolidates and improves search UX.
How It Works
1. Embedder
BERT-based, fine-tuned per platform vocabulary. Same architecture as product-categorization. Multilingual support for cross-language deduplication.
2. Attribute matching
In parallel with semantic similarity: structured attribute matching. Year, brand, model, location, dimensions, when explicit, these are strong signals.
3. Image-based similarity (optional)
For listings with photos, image embeddings add another similarity signal. Real-estate, used-car, and jewelry verticals benefit most.
4. Confidence-based routing
High confidence (semantic, attribute and image signals agree): auto-action. Medium: moderator review. Low: keep distinct.
5. Continuous improvement
Moderator decisions feed Datapipe-orchestrated retraining. New evasion patterns (intentional misspellings, attribute manipulation, paraphrasing) get caught faster.
Stack
BERT-based embedder, vector indexes for similarity search, attribute matching logic, optional image embedder (CLIP-class), Datapipe, moderator workplace.
What You Need to Make This Work
Data. Sample listings, historical duplicate decisions if available.
Integrations. Listing ingestion (upload, API). Moderator workspace. Action endpoint (merge, hide, flag).
Hardware. Cloud or on-prem.
Team. Product / trust-and-safety lead. Moderators for edge cases.
Implementation Roadmap
1. Test (4-6 weeks)
Embed historical listings. Validate detection on labeled duplicates. Output: working detection with measured precision and recall.
2. Pilot (2-3 months)
Production deployment for one vertical / one platform section. Wire up moderator workflow. Tune confidence thresholds. Output: working production deployment with documented business outcomes.
3. Production (3-6 months)
Full rollout. Continuous retraining.
Keep in Mind
- Precision vs recall trade-off. High precision auto-flags fewer but is more confident. High recall catches more but produces more false positives. Configure for your operation.
- Adversarial users adapt. Sellers who want to evade deduplication will paraphrase / restructure listings. Plan for continuous retraining as evasion patterns evolve.
- Image deduplication is harder than text. Lighting, angle, cropping vary even for the same physical item. We use ML-based image similarity, which goes beyond exact matching.
- Real-estate has unique quirks. Multiple agents legitimately list the same property. The “duplicate” might be the right outcome (consolidate listings) or the wrong one (different commission terms). We work through this per platform.
- Multi-language listings need care. Cross-language deduplication is possible with multilingual embeddings; per-language patterns differ.
FAQ
Can this work cross-language?
Yes with multilingual embedders. We’ve shipped cross-language deduplication for marketplaces operating in multiple countries.
What about image-based duplicates (same item, different photos)?
Yes, image embedding adds another similarity signal. Particularly useful for used-car, real-estate, and jewelry verticals.
How does this compare to off-the-shelf duplicate detection?
Most ATS / marketplace platforms have basic duplicate detection (string matching). Our approach is the right choice when you need: semantic understanding of paraphrased listings, cross-language detection, image-based similarity, or fully customizable confidence thresholds.
Can this detect duplicate user accounts as well as listings?
Different problem (identity matching) but adjacent architecture. We’ve engaged but it’s a different scope.
Ready to Discuss?
If you operate a marketplace where duplicates are a real quality issue, this is a focused pilot. We’ll walk through your listing types and your current process, and tell you what to expect.
