Use GPT-4 on millions of records without the credit card surprise

The Problem

Running a large language model at scale produces bills that surprise CTOs. A large language model (LLM) is the engine behind GPT-4 and Claude: you send text in, it sends text back. The default setup fires one request per record and waits for the reply. That works in a demo. In production, across millions of records, the monthly invoice can run into eye-watering numbers.

Most teams find this out the hard way. The pilot runs the model on a sample and looks cheap. The production version runs it on every record. The bill arrives, and the question becomes: keep running, or rip it out?

There is a third path. The patterns that make a model affordable at scale have been around for years. Most teams skip them because the demo worked and nobody planned for the bill.

Caching, deduplication, batching and model routing are plain engineering. None of them need research-grade machine learning. The Datapipe framework we open-sourced makes them straightforward to put in place.

What the Solution Does

This is a cost-optimization engagement for LLM workloads. It cuts the API bill through six steps.

1. Audit. We analyze your current LLM usage and find what drives the cost.
2. Dedupe. Many requests repeat: identical input, model and prompt. We cache the answer and stop paying twice.
3. Cache. Inputs with nearly identical meaning can share an answer (semantic caching: it matches by meaning, so the wording can differ). This saves more, with care.
4. Batch. Many calls bundle naturally. Batch APIs cost less per call and run faster.
5. Route. Easy cases go to a cheaper, smaller model. The expensive model handles the hard ones.
6. Process incrementally. When upstream data changes, only the affected calls re-run.

Where It Fits

This makes sense if you:

• Run LLM workloads at meaningful volume: roughly over 100k API calls a month, or over €1k a month in spend.
• Have data that flows through the model in predictable patterns, such as categorization, summarization or classification.
• Are willing to invest some engineering for ongoing cost savings.
• Want to grow LLM usage while your API spend grows more slowly.

This is probably not the right time if you:

• Run the model at small scale, where the engineering costs more than it saves.
• Use the model only for live, user-facing queries, where there is little to dedupe or cache.
• Cannot tolerate slightly stale answers, since semantic caching brings edge cases.

Business Value

Lower bills. We typically see a 40-80% cut in LLM API costs across the workloads we audit. The engineering usually pays for itself within 1-3 months.

Faster responses. A cache hit returns the answer around 10-100 times faster than a fresh call, so user-facing screens feel quicker.

Room to grow. Teams that could only afford the model on 10% of records can now run it on all of them.

A cleaner codebase. A cost audit often turns up broken or redundant code paths, so the cleanup pays off beyond the bill.

How It Works

The architecture below is our standard pattern for running LLM workloads through Datapipe cost-effectively.

1. Audit phase

We read your LLM usage logs and find the waste: duplicate calls, retry storms, expensive prompts, needless high-temperature variations. This step alone often surfaces around 30-50% of the cost as savings you can take before any engineering.

2. Hash-based deduplication

For each call we hash the combination of input, model, prompt and parameters. If we have made that exact call before, we return the stored answer and skip a second charge. The cache lives in your data infrastructure, so it survives restarts.

3. Semantic caching (optional)

For some workloads, inputs with nearly identical meaning can share an answer. We use embeddings (numeric fingerprints of meaning) and a similarity threshold to spot near-duplicates and serve an approximate answer. Use it carefully: it adds complexity and edge cases.

4. Batch APIs

The OpenAI Batch API, Anthropic batches and similar typically cost around 50% less than real-time calls. They suit offline work well: data labeling, content generation, classification at warehouse scale.

5. Model routing

Not every call needs GPT-4. We add a classifier, or plain rules, to route the work. Easy cases go to a cheaper model such as GPT-4-mini, Claude Haiku or an open-source model. The hard cases still get the expensive one.

6. Incremental processing via Datapipe

This is the piece that matters most. When upstream data changes, Datapipe recomputes only what that change touched. Without it, every data refresh re-runs the model over everything. With it, only the changed records go back through the model.

That is what makes running a model over millions of records affordable.

Stack

Datapipe (open-source) handles incremental processing and orchestration. A persistent cache (Redis, PostgreSQL or S3) stores answers. Routing logic, rule-based or classifier-based, picks the model. Monitoring dashboards track spend.

What You Need to Make This Work

Data. Access to LLM usage logs.

Integrations. Read/write access to your LLM infrastructure. Cache storage.

Hardware. Cloud-based.

Team. Engineering lead. Data engineer.

Implementation Roadmap

1. Audit (1-2 weeks)

We analyze current usage. Output: a written report with the cost breakdown and the savings you can take right away.

2. Initial optimization (2-4 weeks)

We add deduplication, caching and the obvious wins. Output: a working setup with the cost reduction measured.

3. Architecture overhaul (1-2 months, optional)

We add model routing, batch APIs and incremental processing. Output: a production-grade cost-optimization architecture.

4. Monitoring

Ongoing cost dashboards. Alerts on regressions. Quarterly reviews as usage patterns evolve.

Keep in Mind

• Semantic caching has edge cases. Two similar-looking queries can have meaningfully different correct answers. Use semantic cache cautiously and with quality monitoring.
• Batch APIs introduce latency. Real-time workloads can’t use batch. We classify workloads by latency requirement.
• Model routing requires accuracy validation. Routing to a smaller model can degrade quality on some inputs. Measure carefully.
• Cache invalidation is real. When the prompt changes or the model upgrades, cache may need flushing. Plan for this operationally.
• Don’t over-engineer. A plain hash-based cache and basic deduplication capture most of the savings. Pushing harder gives diminishing returns.

FAQ

Does this work with Claude / Gemini / open-source LLMs?

Yes. The pattern is provider-agnostic. Datapipe handles any LLM provider via adapters.

How does this compare to LLM gateway products (Helicone, Portkey, etc.)?

Commercial gateways do some of this, mainly caching and observability. Our approach fits when you need it tied into your existing data pipeline, fully self-hosted, or with custom routing logic. It also fits when you want savings beyond standard caching.

Can this work for RAG / agent workloads?

Yes. Dedupe and incremental processing especially apply to RAG. We’ve shipped this pattern in knowledge-base-assistant deployments.

How much does the engineering cost?

Typically 2-6 weeks of work. At the scales where this matters, the savings usually pay it back within 1-3 months.

Ready to Discuss?

If your LLM API bills are a real budget concern and the setup is naive, with fire-and-forget calls, this is fast-payback engineering. We will walk through your usage and your bills, and tell you what to expect.

Discuss your project