Stop asking 'did the campaign work?' and measure what would have happened without it
Causal inference and incrementality testing. Measure the real lift from your campaigns, separated from what would have happened anyway. This is the right way to answer 'did this work?' for marketing and product decisions.
Discuss a pilot →Quick facts
- Business size
- Mid-market and enterprise, operations running more than 5 measurable campaigns per quarter
- Timeline
- 4-6 weeks test, 2-3 months pilot, 4-6 months production
- Budget range
- Pilot from around €30k.
- Hardware
- Cloud-based; pairs with marketing-attribution or CDP.
- Data needed
- Historical campaign data, transaction history, control or holdout group definitions.
- Evolution
-
- Genesis
- Custom-built
- Product
- Commodity
No product gives you this. We assemble and train it around you.
What this scale means
Further right means more proven and cheaper. Further left means newer and riskier. Here is the test for each step.
- Commodity
- You could get the result yourself from a ready service, with almost no work. We rarely take these on.
- Product
- A vendor already sells this result turnkey, like shelf recognition from Trax or document reading from ABBYY. If one of them fits you, use it. You come to us when it does not: when it has to run on your own servers, cost less, or fit systems the product cannot reach.
- Custom-built
- Vendors sell only the parts. A tool like Tableau hands you charts, but the dashboards and metrics for your business still have to be built. That build is the work, and that is us.
- Genesis
- The approach exists but does not work reliably yet. You are betting on it maturing, so it costs more and carries more risk.
Expected outcomes
The Problem
“Did the campaign work?” is the most-asked, least-answered marketing question. Money rides on the answer, and the usual answer is wrong. Platform reports show clicks and attributed conversions, but most attributed conversions would have happened anyway: the customer was going to buy. Quarterly correlation analyses confuse “things that happened together” with “things that caused each other”.
The real answer to “did it work?” is incrementality: the lift caused by the campaign that would not have occurred otherwise. This is causal inference territory. Methods like geo-experiments, holdout-group designs, synthetic controls, and difference-in-differences are how serious organizations measure marketing.
Most marketing operations skip causal methods. They are harder than reading platform dashboards, they require statistical understanding, and they need data infrastructure. The result is plain: budget keeps flowing to channels that add nothing.
What the Solution Does
You learn which campaigns actually move the business, so you can stop funding the ones that do not. The system is a causal-measurement layer on top of your marketing-attribution and CDP infrastructure (the systems that track ad touchpoints and customer records).
- Experiment design. Geo-experiments, holdout groups, or paired-market designs, depending on what is feasible.
- Pre-experiment power analysis. How much budget and time it takes to detect the expected lift.
- Measurement. Proper statistical analysis (synthetic controls, difference-in-differences, regression discontinuity).
- Reporting. Incrementality estimates with confidence intervals, separated from platform self-reports.
- Strategy feedback. Campaigns that fail incrementality tests get re-evaluated for budget.
Where It Fits
This makes sense if you…
- Run more than 5 measurable marketing campaigns per quarter.
- Have meaningful spend that warrants rigorous measurement.
- Have data infrastructure that can support holdout or geo-experiment designs.
- Have analytics or leadership willing to act on incrementality findings.
This is probably not the right time if you…
- Run small campaigns where platform self-reports are good enough.
- Cannot accept the constraint of holdouts, where some markets do not get the campaign.
- Lack the data infrastructure for the underlying analytics.
Business Value
Spend efficiency. Campaigns that fail incrementality tests get reduced or killed. Typical spend reallocation is around 15-30%, depending on how much current spend is wasted.
Confidence in strategic decisions. “Should we keep running brand campaigns?” stops being opinion and becomes measurement.
Brand-campaign measurement. Brand and awareness campaigns are notoriously hard to attribute by conversions. Incrementality methods (geo-experiments specifically) make them measurable.
Faster decisions. Quarterly correlation reviews give way to run-the-experiment, get-the-answer. Marketing iteration speeds up.
How It Works
1. Experiment design
Different methods fit different campaigns:
- Holdout group: split customers into treatment and control. Most rigorous, and it requires holding some customers out.
- Geo-experiment: split markets into treatment and control. Standard for above-the-line and brand campaigns.
- Paired-market: match similar markets, treat one of each pair. Tighter than random splits for a small sample.
- Synthetic control: use historical pre-treatment data to estimate the counterfactual (what would have happened with no campaign).
- Regression discontinuity / difference-in-differences: when neither holdout nor geo-split is feasible.
2. Power analysis
Before running, we calculate one thing: given the expected lift size, how many units, how much time, and how much budget it takes to detect significance.
3. Run
The campaign launches per design. Monitoring keeps the holdout from being contaminated by spillover.
4. Analysis
Standard causal-inference statistics. We report the point estimate of lift with a confidence interval.
5. Strategic feedback
Incrementality findings inform next-quarter budget allocation. Campaigns that do not pass get re-scoped or cut.
Stack
Python (statsmodels, causalimpact, dowhy, econml), with R for some specialized methods. Datapipe runs the data pipeline, and Metabase or PowerBI shows the dashboards. This often integrates with marketing-attribution and customer-data-platform infrastructure.
What You Need to Make This Work
Data. Historical campaign performance, transaction data, and the ability to define control groups.
Integrations. Read access to your data warehouse. Optionally, the ability to control campaign delivery, for clean holdouts.
Hardware. Cloud-based.
Team. A marketing analytics lead. Marketing leadership willing to accept incrementality findings even when they contradict platform reports. A data engineer.
Implementation Roadmap
1. Test (4-6 weeks)
Pick one campaign for analysis. Design the experiment. Run the analysis on a historical or current campaign. Output: a written analysis with an incrementality estimate and recommendations.
2. Pilot (2-3 months)
Roll out incrementality testing to priority campaigns. Build dashboards comparing platform-reported lift against incrementality-estimated lift. Output: a working testing program with documented business impact.
3. Production (4-6 months)
Incrementality testing becomes standard practice for major campaigns. Marketing-strategy reviews include incrementality findings.
Keep in Mind
- Causal inference requires statistical rigor. Sloppy analysis produces sloppy claims. The methodology matters more than how pretty the dashboard is.
- Holdouts cost money. A 10% holdout means 10% of the spend’s intended audience did not see the campaign. It is worth it for rigorous measurement, and some teams resist.
- Spillover is real. A national campaign with a regional holdout has cross-region effects. We design with spillover in mind.
- Platform reports will not match incrementality estimates. Expect substantial gaps. The platforms over-credit themselves; incrementality is what actually happened.
- Power analysis is critical. Underpowered experiments produce “no detected effect”, which gets misread as “no effect”. We size carefully.
- Some campaigns are too small to measure. Below a certain spend or impact, incrementality testing is not worth the experimental cost.
FAQ
Synthetic-controls or randomized?
Randomized when feasible. Synthetic controls when randomization is not possible: campaigns already running, geographic dependencies, and similar cases.
How does this differ from Google’s or Meta’s “lift studies”?
Platform-run lift studies are useful, and they measure their own platform’s lift in isolation. Cross-channel incrementality requires data from outside any single platform.
Is this just A/B testing?
It is related and broader. A/B testing typically tests product or UX variations on existing users. Causal-impact methods cover marketing-channel measurement, geo-experiments, and other contexts where plain user-level A/B is not feasible.
Can this work for brand campaigns?
Yes, geo-experiments specifically. Geographically-split brand campaigns let you measure incrementality on lower-funnel metrics in test versus control markets.
Ready to Discuss?
If you spend meaningfully on marketing and “did it work?” is currently an opinion-driven debate, this is a rigorous infrastructure investment. We will walk through your campaigns, your data infrastructure, and your decision cadence, and tell you what to expect.
