The MLOps platform we built because Airflow and Prefect didn't fit ML workloads

The Problem

Most ML / data infrastructure stacks were built on tools designed for periodic batch processing, Airflow, Prefect, Dagster. They work fine for “run this pipeline every morning”. They struggle with the ML reality: data arrives continuously, models need to retrain when underlying data changes, and reprocessing all data every time is too expensive at scale.

The result is operations where models go stale because retraining is expensive, dashboards lag because reprocessing takes hours, and engineering teams spend their time managing pipeline failures. The new capabilities they should be building wait.

We built Datapipe, our open-source ETL / MLOps framework, because we needed something that handled incremental processing and dependency tracking natively. Every project we ship runs on it: cargo inspection, Brickit visual search, ECOBOT recycling, document data extraction, cost-effective LLM, and many others.

What the Solution Does

A production MLOps platform built on Datapipe.

1. Incremental processing, when data changes upstream, only the affected downstream computations rerun.
2. Dependency tracking, explicit data dependencies; the framework knows what depends on what.
3. Continuous retraining, model training tied to data changes; retraining happens incrementally.
4. State recovery, pipelines resume from where they failed, so no full rerun is needed.
5. Chain-of-transforms, clear composition of data transformations.
6. Open-source, github.com/epoch8/datapipe, owned by you, with no vendor lock-in.

Where It Fits

This makes sense if you…

• Operate production ML / data workflows where reprocessing cost is real
• Have data that changes continuously and pipelines that should reflect changes quickly
• Want vendor-independent infrastructure (open-source, compared with Databricks, Vertex, SageMaker)
• Have an engineering team capable of operating data infrastructure

This is probably not the right time if you…

• Are at small scale where plain Airflow or cron works
• Want a fully-managed SaaS solution (Databricks, Vertex, etc.): we deploy Datapipe, and it is still infrastructure to operate
• Have no data engineering team: Datapipe is a framework that needs engineers, with no no-code product to fall back on

Business Value

Pipeline run-time. Typically 5-10× faster than full reruns, depending on how much of the data actually changed. This is the value of incremental processing.

Data freshness. ML inference pipelines that depended on daily or weekly batch updates now reflect data changes in minutes.

Compute cost. Typically a 40-70% reduction. Most ML pipelines reprocess data that hasn’t changed, and Datapipe skips that work.

Engineering velocity. Deploying new model versions, new data sources, new pipeline stages takes hours, down from days.

Vendor independence. Open-source means no vendor lock-in. Critical for organizations that want infrastructure ownership.

How It Works

The framework below is Datapipe, our incremental data-processing engine.

1. Define transforms

Each step in a pipeline is a function: input in, output out. Datapipe handles the orchestration.

2. Dependency tracking

The framework knows that “feature X depends on raw data Y”. When Y changes, X needs recomputation. When Y is unchanged, X doesn’t.

3. Granular state

Datapipe tracks state at row-level (or whatever granularity makes sense). Adding one new row triggers minimal downstream work.

4. State recovery

If a pipeline fails mid-run, it resumes from the last completed checkpoint. No re-running from scratch.

5. ML-specific patterns

Built for ML workflows: incremental embedding recomputation, model retraining triggered by data changes, validation-vs-training set separation handled cleanly.

6. Integration with the rest of the stack

Datapipe works with whatever data stores you have: PostgreSQL, ClickHouse, S3, Redis, etc. Kubernetes-friendly deployment.

Stack

Datapipe (open-source, Python), Kubernetes for orchestration (when needed), state storage (PostgreSQL or similar), monitoring via Metabase / Grafana, integration with model registries.

What You Need to Make This Work

Data. Your current data / ML pipelines. Pain points (what’s slow, brittle, expensive).

Integrations. Existing data infrastructure.

Hardware. Cloud or on-prem. Kubernetes preferred.

Team. Data / ML engineer who’ll own the platform.

Implementation Roadmap

1. Initial (4-8 weeks)

Audit current pipelines. Identify high-ROI migration targets (slow or expensive pipelines). Migrate the first 2-3 pipelines to Datapipe. Output: working Datapipe deployment and measured improvement on migrated pipelines.

2. Iterative buildout

Migrate additional pipelines as priority dictates. Add monitoring. Train your team on Datapipe operations.

3. Ongoing

Your team owns day-to-day. We stay on for advanced patterns, performance optimization, and new feature requests.

Keep in Mind

• Datapipe is a framework. It requires engineers to operate, so it does not suit organizations expecting a fully-managed UX.
• Migration is real work. Existing pipelines need rewriting against Datapipe’s model. We migrate incrementally.
• Granularity tuning matters. Wrong granularity hurts performance: too fine adds overhead, too coarse misses the value. We tune during pilot.
• Open-source means open-source. Datapipe is on GitHub. You can use it without us. We add value through expertise, integration, and new feature development.
• It is not Spark or Hadoop. Different architecture. Datapipe is for incremental ML and data workflows. Spark is for big-data batch processing.
• The framework is opinionated. Some patterns are easy. Some require thinking in Datapipe’s model. Worth the investment for ML workloads, possibly overhead for plain ETL.

FAQ

How does this compare to Airflow / Prefect / Dagster?

Airflow et al. orchestrate batch jobs. Datapipe handles incremental processing with row-level dependency tracking. Different design point.

How does this compare to Spark / Databricks?

Spark is for big-data batch processing. Datapipe is for incremental processing where most data hasn’t changed. Different scale and workload.

How does this compare to MLflow / Vertex / SageMaker?

Those handle model lifecycle (training, registry, serving). Datapipe handles the data pipeline that feeds those. Often used together.

Self-host or managed?

Open-source, self-host. We don’t operate Datapipe-as-a-service.

Is there commercial support?

We provide expertise, integration, and new feature development as engagements. The framework itself is free.

Ready to Discuss?

If you operate production ML / data workflows where pipeline performance and incremental processing are real concerns, this is foundational infrastructure investment. We’ll walk through your current pain points, and tell you what to expect.

Discuss your project