A counterfactual forecast is a model's prediction of what the treated markets would have done had the campaign not run. The measured effect is the gap between actual and forecast, so the estimate is only as good as the forecast behind it.
Every geo test report contains a chart with a solid line for what happened and a dotted line for what would have happened otherwise. The dotted line is the counterfactual forecast, and it is a model output rather than an observation — the counterfactual itself is unobservable, so it has to be predicted. The measured effect is the area between the two lines, which makes the forecast the load-bearing part of the whole analysis.
Different methods build it differently. A matched-market design uses the control markets' actual trajectory directly. Synthetic control constructs a weighted blend of untreated markets chosen to track the treated ones over the pre-period. CausalImpact fits a Bayesian time-series model using control markets as predictors and projects it forward. Each is answering the same question with different machinery, and each rests on the same assumption: that the relationship observed before the campaign continues to hold during it.
That assumption is what has to be validated, and pre-period fit alone does not validate it. A model can fit the pre-period beautifully by overfitting — enough donor markets and enough flexibility will reproduce any historical series — and then forecast badly. The meaningful test is holding out a window: fit on data ending before some period the campaign did not touch, forecast into it, and compare against what actually happened. A model that cannot predict a quiet month it did not see should not be trusted to predict a counterfactual.
The forecast also has uncertainty that is frequently discarded in reporting. A prediction interval widens the further out it projects, so a twelve-week campaign has a much less certain counterfactual at week twelve than at week one. Reporting a single effect number without the interval implied by that widening is the most common way a geo result is overstated — the point estimate is often reasonable and the confidence attached to it is not.
The practical failure to watch for is something changing in the control markets rather than the treated ones. If a competitor launches nationally mid-test, both sets are affected and the relationship may hold; if it launches only in the control regions, the forecast is wrong in a direction that has nothing to do with the campaign. Monitoring the control markets during the test, not just at the end, is what makes that detectable while there is still time to respond.
The estimator, and the validation that distinguishes a trustworthy forecast from a well-fitted one.
lift = Σ ( actual_t − forecast_t ) over the test periodThe area between the lines. Every geo method computes this; they differ in how the forecast is built.
the pre-period relationship holds through the test periodUntestable during the test itself, which is why holdout validation beforehand matters.
fit on data ending at T−k, forecast the k weeks to T, comparePre-period fit alone is not evidence — an overfitted model fits perfectly and forecasts badly.
prediction interval grows with horizonA twelve-week counterfactual is far less certain at week twelve — see the correlation calculator for the pre-period fit check.
A geo test's counterfactual forecast is validated two ways before the campaign result is read: by pre-period fit, and by a holdout test on the eight weeks immediately before the campaign started.
An excellent in-sample fit and a forecast that runs 4.2% high out of sample — which would have understated the campaign effect by nearly half.
The first two rows would satisfy most reviewers: R² of 0.982 and in-sample error of 1.4% look like a model that has learned the relationship. The holdout says otherwise. Forecasting eight weeks it had not seen, the model overshot by 4.2% on average, which means it would have predicted a higher baseline than reality and made the campaign look smaller than it was. Correcting for that bias moves the estimate from +5.9% to around +10.1%, though with much wider uncertainty, and the honest report says the effect is somewhere in that region rather than picking either number. The general lesson is the gap between the second and third rows: in-sample fit measures how well a model reproduces data it has seen, which is nearly free with enough donor markets, and says very little about forecasting. Any geo result should come with a holdout validation, and a model that fails one should be rebuilt rather than corrected.
When your advertising efforts are up and sales are increasing, that’s great news, but it doesn't necessarily mean the ads are the reason. Here are three popular tools to measure the lift of a campaign.
Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.