What is difference-in-differences?

You launch something in six markets and sales rise 8%. The naive reading treats zero as the counterfactual, which is almost never right — sales might have risen 5% anyway on seasonality alone. Difference-in-differences fixes that by borrowing a comparison: look at markets that received nothing, measure how much *they* changed over the same window, and subtract. What remains is the change specific to the treated group, and it is a much better estimate of the effect than either single difference.

The name is literal. The first difference is before-versus-after within the treated group, which removes everything permanently distinctive about those markets — their size, their baseline conversion, their customer mix. The second difference is the same before-versus-after in the untreated group, which captures everything that happened to everyone over that period: the season, the economy, a site-wide change, a competitor's promotion. Subtracting the second from the first removes both classes of nuisance at once, which is why the design is so widely used with observational data.

Everything rests on one assumption, and it is worth stating in its exact form: in the absence of treatment, the two groups would have moved in parallel. Not that they were at the same level — the design explicitly allows different baselines — but that their *trends* would have matched. This cannot be verified, because it is a claim about a period that did not happen. What you can do is check whether the trends were parallel before the intervention, over as long a pre-period as you have, which is evidence for the assumption rather than proof of it.

Two failures break it in practice. Selection into treatment: if the six markets were chosen because they were underperforming, or because a regional manager pushed for them, they were already on a different trajectory and the parallel trends assumption is false in a direction you cannot see. And anticipation: if anyone changed behaviour before the official start date — a pre-launch push, retailers stocking up — the pre-period is contaminated and the effect is understated. When treatment assignment is non-random in a way you cannot argue away, synthetic control is usually the better tool, because it builds a comparison group weighted to match the treated unit's history rather than assuming an existing group will do.

Two technical points that bite. Inference on repeated observations of the same units is not standard: outcomes within a market are correlated over time, so ordinary standard errors are badly understated and the errors must be clustered at the treated unit. And with units treated at different times, the classic two-way fixed-effects regression can be biased — sometimes even carrying the wrong sign — because it uses already-treated units as controls for later-treated ones. Modern estimators from the staggered-adoption literature fix this and should be the default whenever the rollout was phased.

A four-week out-of-home campaign runs in eight metro areas. Twelve comparable metros receive nothing. Weekly revenue per 100,000 population is measured for twelve weeks before and four weeks during.

The naive 13% would have credited the campaign with the whole seasonal rise that lifted every metro; the honest figure is roughly 5.5%. Before believing it, plot the twelve pre-period weeks for both groups: if the treated metros were already pulling away, the assumption is broken and the estimate is inflated by whatever the divergence was. Two further checks earn their keep. The interval is built on clustered standard errors, and with only eight treated clusters it is wide — unclustered errors here would have been perhaps a third of the width and badly overconfident. And a placebo test, running the same estimator on a period before the campaign, should return an effect indistinguishable from zero; if it does not, the design is picking up something other than the campaign.