Process terabytes daily without the cloud bill growing every month
Incremental recomputation, change tracking, and deduplication. The engineering pattern that keeps recurring data-processing cost flat as the dataset grows, built on our open-source Datapipe framework.
Discuss a pilot →
Quick facts
- Business size
- Any operation reprocessing large datasets on a schedule
- Timeline
- 2-4 weeks audit and initial optimization, 1-3 months for a full incremental rebuild
- Budget range
- Engagement from EUR 20k. Often pays back in saved cloud bills within months.
- Hardware
- Cloud-based (GCP, AWS, or Azure).
- Data needed
- Access to current pipeline code, cloud billing, and source data schemas.
- Evolution
-
- Genesis
- Custom-built
- Product
- Commodity
No product gives you this. We assemble and train it around you.
What this scale means
Further right means more proven and cheaper. Further left means newer and riskier. Here is the test for each step.
- Commodity
- You could get the result yourself from a ready service, with almost no work. We rarely take these on.
- Product
- A vendor already sells this result turnkey, like shelf recognition from Trax or document reading from ABBYY. If one of them fits you, use it. You come to us when it does not: when it has to run on your own servers, cost less, or fit systems the product cannot reach.
- Custom-built
- Vendors sell only the parts. A tool like Tableau hands you charts, but the dashboards and metrics for your business still have to be built. That build is the work, and that is us.
- Genesis
- The approach exists but does not work reliably yet. You are betting on it maturing, so it costs more and carries more risk.
Expected outcomes
The Problem
Broadcast logs, transaction streams, sensor feeds, event data. Many businesses run analytics on datasets that are large and grow every day. The naive pipeline recomputes the full history on every run: rebuild every metric, every data mart, every report, from scratch, daily.
This works on a sample. It works in the first months of production. Then the dataset crosses a threshold and the daily job either runs out of compute or runs up a cloud bill that no one budgeted for. As the history keeps growing, the cost of recomputing it grows with it, and eventually the job stops finishing at all.
The fix is not a bigger machine. The fix is to stop recomputing data that did not change. When only a fraction of records are new or updated each day, recomputing the whole history is wasted work, and on metered cloud compute, wasted work is a line item.
What the Solution Does
A cost-optimization engagement for batch and scheduled data pipelines.
- Audit. Analyze the current pipeline and cloud billing. Find what gets recomputed needlessly.
- Track changes. Identify which source records are new or changed since the last run.
- Recompute incrementally. Recalculate only the affected metrics and marts, leave the rest untouched.
- Deduplicate. Drop duplicate and redundant work at load time.
- Optimize storage and queries. Restructure marts so the warehouse scans less data per query.
- Export cleanly. Push results to the databases and buckets downstream systems already use.
Where It Fits
This makes sense if you…
- Reprocess a large dataset on a schedule (daily or more often) and the dataset keeps growing
- See cloud compute or warehouse bills rising faster than the business value
- Need fresher metrics than a once-a-day full rebuild can deliver
- Have a pipeline that already failed, or is about to, under its own data volume
This is probably not the right time if you…
- Work with small or static datasets where a full rebuild is cheap
- Run a one-off analysis, with no recurring pipeline
- Cannot get access to the source data schemas or the current pipeline code
Business Value
Lower recurring cost. Incremental processing typically cuts recurring compute by a large margin versus full reprocessing. The savings recur every run, so the engineering investment usually pays back in months.
A predictable bill. Cost tracks the volume of change. The size of the dataset stops driving it. The monthly budget stays roughly flat as history accumulates.
Fresher data. When a run no longer has to touch the whole history, it can run far more often. Metrics refresh in minutes, where a full rebuild ran once a day.
Headroom. Pipelines that were failing under their own volume keep running, and keep room to grow.
How It Works
1. Audit phase
Read the current pipeline and the cloud billing together. Find the expensive steps: full-table scans, daily rebuilds of slow-changing data, duplicate loads, oversized marts. This alone often surfaces immediate savings before any rebuild.
2. Change tracking
For each source, determine what is new or updated since the last successful run. This is the core of incremental processing: a reliable record of what changed.
3. Incremental recomputation via Datapipe
Datapipe, our open-source orchestration framework, recomputes only what depends on changed inputs. When a slice of source data updates, only the affected metrics and marts recalculate. The rest is left as-is. This is the piece that makes processing-on-every-record economical.
4. Deduplication at load
Many loads carry duplicate or already-seen records. Deduplicating on the way into the warehouse avoids paying to process a row twice.
5. Warehouse and mart optimization
Restructure reporting marts so common queries scan less data. On usage-billed warehouses such as BigQuery, less scanned data is a direct cost reduction.
6. Export to downstream systems
Write results to the databases and storage downstream systems already read, for example a client ClickHouse instance or an output bucket, so nothing else has to change.
Stack
Datapipe (open-source) for incremental processing and orchestration, a cloud data warehouse (BigQuery, Snowflake, or similar), object storage for staging and exports, and managed compute (for example Cloud Run) for the processing steps.
What You Need to Make This Work
Data. Access to source data schemas and a way to tell what changed between runs.
Integrations. Read access to the current pipeline and cloud billing. Write access to the target warehouse and downstream stores.
Hardware. Cloud-based.
Team. An engineering lead on your side and a data engineer. We bring the pipeline work.
Implementation Roadmap
1. Audit (1-2 weeks)
Analyze current processing and billing. Output: a written report with a cost breakdown and immediate-savings opportunities.
2. Initial optimization (2-4 weeks)
Implement change tracking and dedup, remove the obvious waste. Output: a working pipeline with measured cost reduction.
3. Incremental rebuild (1-3 months, optional)
Restructure the pipeline around incremental recomputation end to end. Output: a production pipeline whose cost tracks change volume, so history size no longer drives the bill.
4. Monitoring
Ongoing cost and freshness dashboards, alerts on regressions, and periodic reviews as data volume grows.
Keep in Mind
- Change tracking has to be correct. If the pipeline misjudges what changed, it can miss updates or recompute too much. Getting this right is the heart of the work.
- Incremental rebuilds add complexity. A full rebuild is easy to reason about. Incremental logic adds moving parts. It pays off at scale and is overkill for small data.
- A one-time backfill can spike the bill. The first incremental run, or a schema change that forces a full recompute, can be expensive. Plan and budget for it.
- Don’t over-engineer. Removing the obvious waste captures most of the savings. Aggressive optimization beyond that yields diminishing returns.
FAQ
Does this work on AWS or Azure, beyond GCP?
Yes. The pattern is cloud-agnostic. Datapipe orchestrates incremental processing against any warehouse and storage.
How is this different from a managed ETL tool?
Managed tools move and transform data. They do not, on their own, decide what to recompute. The savings here come from incremental recomputation and warehouse optimization, which sit on top of whatever moves your data.
We already use dbt. Does this replace it?
No. dbt models transformations; Datapipe decides which of them need to re-run when inputs change. They fit together.
How much does the engineering cost?
The audit and initial optimization are a few weeks. At the scales where this matters, the recurring savings usually pay it back within months.
Ready to Discuss?
If a scheduled pipeline is getting slower and more expensive as your data grows, this is fast-payback engineering. We will look at your pipeline and your cloud bills and tell you what to expect.
