What is CausalImpact?

CausalImpact estimates the effect of an intervention on a time series when a controlled experiment was not run. It fits a Bayesian structural time-series model to the pre-intervention period — decomposing the series into trend, seasonality and the contribution of control series that were not affected — and then projects that model through the intervention period to produce a counterfactual forecast. The gap between actual and forecast is the estimated effect.

Its advantages over a simple before-and-after comparison are substantial. It models trend and seasonality explicitly rather than assuming them away. It uses control series as predictors, so anything affecting both treated and control markets is absorbed. And because it is Bayesian, it returns a credible interval on the cumulative effect rather than a point estimate, with the uncertainty widening appropriately as the forecast extends.

Everything depends on the control series, and this is where implementations go wrong. They must be predictive of the treated series before the intervention, and they must be genuinely unaffected by it. A control region that also received some campaign spend, or a metric downstream of the one being tested, contaminates the counterfactual and biases the effect towards zero. Picking control series by correlation alone will happily select contaminated ones, since a contaminated series correlates well by construction.

The second failure is treating pre-period fit as validation. With enough candidate predictors the model will fit the pre-period almost perfectly and can still forecast badly — the spike-and-slab prior does some variable selection, and it is not a substitute for testing the forecast on held-out data. Running the analysis on a period where nothing happened, and confirming it reports no effect, is the cheapest available sanity check and is skipped remarkably often.

Compared with the alternatives, it sits between a matched-market geo experiment and synthetic control. A randomised geo test is stronger, because assignment rather than a model creates the counterfactual, and is not always possible — the campaign may already have run. Synthetic control constructs a weighted combination of donors with constrained weights; CausalImpact fits a fuller time-series model with regression on the controls. In practice both are applied to similar problems and often agree, which is itself a useful check.

A brand runs a six-week regional television campaign that was never randomised, and analyses it with CausalImpact using twelve untreated regions as control series. Two validation checks are run before the result is accepted.

The placebo check is what makes the headline believable: run on a window where nothing happened, the model correctly found nothing, which is evidence the machinery is not manufacturing effects. The contamination finding is the more instructive part. Two of the twelve control regions sat adjacent to the campaign area and received overspill television, so they were partly treated — and because they were partly treated they tracked the treated series especially well, which is exactly why a correlation-based selection had chosen them. Including them pulled the counterfactual up towards the treated trajectory and understated the effect by three percentage points. The general rule is that control series must be verified as genuinely unexposed on the media plan, not merely as well-correlated. And note the interval width: +2.1% to +14.6% is a wide range, which is honest for a non-randomised six-week analysis and would look much tighter if the forecast uncertainty were dropped.