Pet Search: Visual Matching for Lost and Found Pets

The client lets people post a pet they have lost and people post a pet they have found, in several countries. The match between the two is the service. An incumbent production system already paired posts, so the brief was not to start from nothing but to do it better.

The catalog is large and uneven: around 180,000 posts, each carrying several photos plus the animal type, sex, date and a lost or found label, for roughly 700,000 webp images. Photos are casual phone shots in any lighting and pose, often with more than one animal in frame.

For any lost post we need the most visually similar found posts, and the reverse, narrowed by a search radius of about 150 km and a time window so that a found post is plausible for the loss.

Each photo runs through a YOLO-style segmentation model that finds the animal and returns both a box and an outline mask. The image is cropped to the animal, then passed through a DINO-style embedding model to get a 1024-dimension vector. Within a single post, photos that are near-identical are dropped at a 0.95 similarity threshold so repeated shots do not skew the score. Every vector and its metadata land in Qdrant.

To match a post, each of its images queries the opposite pool in Qdrant for its nearest neighbors. Those neighbors are grouped by post, sorted by cosine similarity, and the top three image pairs are averaged into a single score for that post. Posts are ranked by that score and returned with the animal boxes and masks and the closest photo, so the result can show a preview and blur everything outside the animal.

Choosing the embedder was its own piece of work. We benchmarked more than ten open vision embedders, including DINO-style, SigLIP-style, CLIP and several animal-tuned variants, against location and time filters. A DINO-style model gave the best balance on the full labeled set. The known weak spot is color: the model leans on shape and texture, so two animals of clearly different coat color can still score high.

We compared our pipeline against the incumbent production system head to head. For 894 lost posts we placed our ranked candidates next to the incumbent's and asked GPT-5.2 to pick the better ranking, with temperature 0, a fixed seed and structured output, at about three dollars per full run. Our ranking won 656 posts, the incumbent won 226, and 12 were a tie, so we came out ahead on roughly 74 percent of the decided posts. The margin held for both animals: 398 to 88 on cats and 258 to 138 on dogs.

Measured by coverage, the system surfaces about 70 percent of the good candidates in the top 10 for dogs and 60 percent for cats, rising to 85 and 80 percent in the top 20. Those numbers are deliberately conservative, because the labeling counted anything only slightly off as a non-match.

The whole thing runs on CPU-only hardware, around 8 CPU and 20 GB of RAM, processing a post in roughly five to six seconds, with backups, restore and a documented deploy. A GPU path is available for the one-time bulk load of the existing catalog.