Stop typing data from scans, let an ML pipeline do it

The Problem

Every back-office operation has one recurring task: take a scanned document, read it, type the relevant fields into the ERP. Invoices, delivery notes, purchase orders, contracts, certificates, customs declarations: the document types vary; the workflow is identical.

At small scale it’s tedious. At enterprise scale (tens of thousands of documents per month) it’s expensive and slow. Backlogs build up. Suppliers wait to be paid because their invoice is still in the queue. The team that should be doing financial analysis is doing data entry instead.

Off-the-shelf OCR has been promised to fix this for two decades. The reason it didn’t: real documents are messy. Scans come in tilted, photographed from a phone, with hand-stamps over the fields, with multi-page footers, in document formats nobody anticipated when the OCR rules were written. The brittle nature of template-based extraction means every new vendor breaks the system.

CV-driven detection plus OCR plus continuous fine-tuning is what changes the picture. Detection finds the fields; OCR reads them; continuous fine-tuning means the system adapts to new formats as they appear.

What the Solution Does

A pipeline that takes scanned documents in, produces structured data out, and gets better over time.

1. Ingest: documents arrive via email, file upload, API, or scanner queue.
2. Detect the document: find the document boundaries in the image (phone photos may include hands, backgrounds, etc.).
3. Detect fields: find the key fields within the document (date, total amount, line items, etc.).
4. OCR: read text and numbers from each detected field.
5. Structured output: fields linked to their values, exported via API to your ERP, accounting system, or data warehouse.
6. Continuous improvement: moderators correct edge cases via Label Studio; Datapipe handles incremental retraining.

Where It Fits

This makes sense if you…

• Process more than 1,000 structured paper or scanned documents per month
• Have measurable manual data-entry cost (FTE allocated, or business slowdowns from backlog)
• See evolving document formats from suppliers / customers / agencies
• Have an ERP / accounting system / DB that can consume structured output via API
• Want a pipeline that keeps improving, where a one-time vendor configuration would go stale

This is probably not the right time if you…

• Process a handful of documents per week, manual entry is cheaper
• Process documents with no consistent structure (fully free-form legal text, different problem, see content moderation or knowledge base assistant for those)
• Have only one document format that never changes, template-based commercial OCR may be cheaper
• Cannot integrate with your downstream system, extraction without integration is solving half the problem

Business Value

Throughput. A trained pipeline processes thousands of documents per hour on commodity hardware. Manual entry caps at tens of documents per FTE per day.

Accuracy. Field-extraction accuracy on trained document formats reaches 90-97% in production. That’s per-field, against held-out validation sets. The remaining 3-10% gets flagged for moderator review, so the system does not silently mis-extract.

Adaptability. When a new supplier sends a new invoice format, conventional template-based OCR breaks. Our pipeline includes a continuous fine-tuning loop: new formats get added in 2-4 weeks via Datapipe, without rebuilding the pipeline.

Backlog elimination. Most teams running this for the first time discover the real bottleneck: the work ran at human-attention rates. With the pipeline absorbing 90% of documents automatically, the human team handles only exceptions; backlogs shrink fast.

How It Works

The architecture below is what shipped for invoice extraction (the canonical case study). It applies equally to delivery notes, purchase orders, contracts, certificates, and similar structured documents.

1. Document detection (YOLOv5)

The first step finds the document itself in the image, important because photos often include hands, background objects, or document edges that aren’t relevant. YOLOv5 detects the document boundaries and the rotation angle (0°, 90°, 180°, 270°). The system isolates the document and rotates it to canonical orientation before further processing.

2. Field detection (YOLOv5, multi-stage)

The image gets divided into sub-pictures (smaller image regions). Each sub-picture goes through field detection, locating the date, total amount, line items, vendor name, and so on. Sub-pictures improve accuracy because models work better on smaller, focused regions than on full-document context.

Multi-stage is the right architecture here: a one-shot model that tries to find everything at once works on clean documents but fails on edge cases. The two-stage detection with per-region analysis is more accurate, at the cost of being a bit slower.

3. OCR (Google Cloud Vision OCR or self-hosted alternatives)

For each detected field, OCR reads the text or numbers inside. We typically use Google Cloud Vision OCR for the broadest accuracy out of the box; for clients with data-residency requirements we use self-hosted alternatives (Tesseract with fine-tuning, or other open-source options).

4. Field-value linking

Detected fields linked to their extracted values. The output is structured: { "invoice_number": "INV-2026-0042", "date": "2026-04-15", "total": 12450.00, "line_items": [...] } and similar.

5. Continuous fine-tuning (Datapipe)

The whole pipeline is built on Datapipe. Moderators correct extraction errors via Label Studio. New labelled data feeds a frozen dataset. When enough new data accumulates, the model retrains automatically. The new model’s metrics are computed on validation data, and if it outperforms the current production model, it gets registered for deployment.

This is what makes new document formats workable. A new vendor’s invoice does not break the system. It goes into the queue for moderation, contributes to the dataset, and triggers a retraining cycle.

6. Production deployment: RabbitMQ and a daemon

New documents arrive in a RabbitMQ queue. A daemon process monitors the queue, processes each message through the pipeline, and sends results to the API. The API is consumed by the client’s downstream system (ERP, accounting, or wherever the structured data is needed).

Stack

YOLOv5 for detection, Google Cloud Vision OCR (default) or self-hosted OCR alternatives, Datapipe for ETL and incremental retraining, Label Studio for annotation, Metabase for metrics dashboards, RabbitMQ for queue processing, Python/FastAPI for the API layer. Cloud or on-prem deployment.

What You Need to Make This Work

Data. 100+ examples per document type for initial training. We can collect from your historical archive or run a data-collection sprint at the start. Reference samples of typical and edge-case formats.

Integrations. Document ingest channel (email, API, scanner queue, file upload). Downstream API for structured output (ERP, accounting, data warehouse). RabbitMQ or equivalent message queue if your processing volume is high enough to need queuing.

Hardware. Cloud or on-prem. CPU sufficient for inference (a single mid-range server handles thousands of documents per hour). GPU helpful for retraining cycles.

Team. A document-ops lead who’ll own the moderation workflow. A moderator pool (yours or ours) for the continuous fine-tuning step, typically 5-15 hours/week for a 10,000 documents/month operation. An IT contact for the downstream integration (~20-40 hours during pilot).

Implementation Roadmap

1. Test (4-6 weeks)

Pick one or two document types (most common, or most painful). Collect labelled examples. Train the first version of the pipeline. Measure baseline accuracy. Output: working pipeline for these document types with measured accuracy, recommendations for pilot scope.

2. Pilot (2-3 months)

Production deployment to one workflow (one document type, one team, or one volume slice). Set up moderation. Build the downstream integration. Measure throughput, accuracy, and team-level impact on backlog. Output: working production deployment with documented business outcomes.

3. Production (4-6 months)

Expand document types, expand to network-wide operation. Continuous fine-tuning runs unattended. Quarterly accuracy review. Your team owns the moderation workflow; we stay on for new document type onboarding and retraining oversight.

Keep in Mind

Where it breaks:

• It’s not 100%. 90-97% on trained types is good. 100% doesn’t happen: exceptions are routed to moderators, by design.
• New formats need data. A document format the system has never seen will be poorly handled until we collect examples and retrain. Plan for onboarding cycles. Immediate support for an arbitrary new format is not realistic.
• Document quality matters. Tilted, faded, partially-cropped, or hand-stamped documents drop accuracy. The system handles common variations; extreme cases get flagged for human review.
• Handwriting is hard. Numeric fields with neat handwriting work; cursive narrative text is much harder. We map out the limits during pilot scope.
• OCR provider choice has cost trade-offs. Google Cloud Vision OCR is excellent but per-page costs add up at high volume. Self-hosted alternatives are cheaper but require fine-tuning for some scripts and document types.
• Compliance constraints affect architecture. Some clients can’t send documents to cloud OCR (GDPR, sector regulations). The pipeline supports fully on-prem deployment in those cases, with the trade-off of needing self-hosted OCR.

FAQ

What document types do you support?

We’ve shipped extraction for invoices, delivery notes, purchase orders, certificates, customs declarations, receipts, and various contracts. Any structured document with consistent layout patterns is tractable. Fully free-form documents (legal narratives, articles) need a different approach.

Can this handle multi-language documents?

Yes. OCR layer handles multilingual content. Field-detection works across languages once trained. Most common: English, Russian, German, French, Spanish in our deployments.

What if our documents have hand-stamps, hand-written annotations, or other clutter?

Hand-stamps and overlay clutter get detected as visual interference, the system either handles them or flags the document for moderator review. We measure tolerance during pilot.

How does this compare to commercial IDP platforms (Hyperscience, Rossum, etc.)?

Commercial IDP platforms are excellent for organizations that want a ready-made product to configure. Our approach fits when you need heavy customization, ongoing model improvement, full on-prem deployment, integration with non-standard downstream systems, or substantially lower per-document cost at very high volume.

Can the system cross-validate against existing master data?

Yes, that’s the related document cross-validation use case. Detected invoice values can be checked against your supplier master, PO database, contract terms, and other source-of-truth systems.

Ready to Discuss?

If you process structured documents at meaningful volume and the manual data-entry cost is a real line item, this is a fast-ROI deployment. We’ll walk through your document types, your downstream systems, and your current workflow, and tell you what to expect from a pilot.

Discuss your project