Real-time product marking recognition on a smartphone

A national mandatory-marking operator runs the track-and-trace system for regulated goods. Every unit of alcohol, tobacco, dairy, and other regulated categories carries a unique Data Matrix code that ties it to that system. The operator's app lets someone point a phone at a product and read the marking straight from the camera.

The hard part is the surface. A Data Matrix on a real product is rarely a clean printout. It sits on foil and glossy film on alcohol and tobacco, wraps around bottles and cans, catches glare, and on some products is punched into metal as a grid of dots.

Photos arrive blurred, partial, or shot at an angle. A general mobile barcode reader handles a flat, well-lit code and misses much of the rest. The operator had been using one and losing too many real-world scans. They needed recognition that holds up on the products people actually scan, running on the phone itself so a scan takes no round trip to a server.

The app turns a phone into a scanner. The user points the camera at a product, and the system works on the live stream, with no tap to scan and no pause to align.

Detection runs first. A lightweight YOLO-like detector, tuned for mobile, finds each marking region in the frame through TensorFlow Lite and returns a bounding box (Bbox), a class (Class), and a confidence score (Score). We chose the lightweight variant over larger ones because it keeps up with the live feed on mid-range Android and iOS phones.

Decoding is the heart of it. The Data Matrix holds the track-and-trace payload, and decoding it off a real product is the hard part. We evaluated the available decoders (Google ML Kit, Apple Vision, libdmtx, ZXing), then forked ZXing and patched it for stability on hard codes. A preprocessing stage sits in front of the decoder: binarization, blur, sharpening, contrast, morphology, and resampling, with settings tuned by Optuna against our own validation sets. Resizing a small or skewed crop before decoding mattered more than any single filter.

Classification covers the rest. Each detected region passes through a MobileNetV2 classifier that names the mark directly: one of about 128 marking classes, covering conformity marks, recycling symbols, handling instructions, ecology certifications, clothing-care icons, and more. There is no hierarchy; the model predicts the 128 classes in one step. The decoded code and the surrounding symbols then merge into a single result per frame.

The whole pipeline runs on the phone. MediaPipe drives the real-time graph and ships as native builds for Android and iOS, OpenCV handles the frame work, and both models and the decoder run on-device with no network call.

From a single photo, the app boxes every marking, decodes the Data Matrix into its track-and-trace code, and lists the product's other markings on screen. The user sees the full breakdown without leaving the app, and it works with no connection because nothing leaves the phone.

Clean, well-lit codes read reliably. The metric we track is the read rate: the share of codes recovered from a photo or video. We have not run the first app version and the latest against one fixed set under fixed conditions, since over four years both the data and the way we score it changed. Over a comparable two-year window the read rate climbed sharply. On live video, alcohol marks went from around 61% to 82% and tobacco from around 47% to 73%. On single images, alcohol rose from around 24% to 91% and tobacco from around 9% to 58% (read rates on our test sets, versions 4.6.4 to 6.2.0, approximate). Dot-peen codes punched into metal stay the hard case, alongside heavy glare and steep angles.

The recognition engine also runs behind an API, so one pipeline serves the phone app and any server-side check that needs it. It has been in production across many release cycles since 2021, with the models, the decoder, and the Android and iOS builds updated as the marking system and the phones change.