
Measuring the true effect of your ads with Geo Testing
Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.

A synthetic control is a counterfactual built by weighting together untreated units so that their combined history closely matches the treated unit's own history before the intervention. The gap that opens up between the two after the intervention is the estimated effect.
Difference-in-differences needs a comparison group whose trend already matches the treated group's. Often none exists — you rolled out in your three biggest cities, or in the one region where the media buy was available, and no untreated region looks anything like them. Synthetic control solves this by constructing the comparison instead of finding it: take all the untreated units, and find the weighted combination of them whose pre-intervention path most closely tracks the treated unit's.
The output is a single fictional unit — 0.4 of one city, 0.3 of another, 0.2 of a third — that behaved like the treated one for as long as you have data before the intervention. After the intervention, it continues to reflect whatever was happening to untreated markets, so the gap between the real unit and its synthetic twin is the estimated effect. The great practical virtue is that the pre-period fit is visible: you can plot the two lines, and everyone in the room can see whether the counterfactual is credible before any effect is quoted.
That transparency is also the method's own honesty check, and it enforces a discipline the alternatives lack. The weights are chosen only on pre-intervention data, so they cannot be tuned to produce a flattering answer. They are usually constrained to be non-negative and sum to one, which forbids extrapolation — the synthetic unit is a genuine blend of real units rather than an arithmetic fantasy — and typically produces a sparse solution using a handful of donors, so you can name which markets are standing in for yours.
It fits situations difference-in-differences handles badly: one or a few treated units, non-random selection into treatment, and a long pre-period. That is exactly the shape of most regional marketing questions, which is why it has become a standard tool for geo experiment analysis and for evaluating campaigns that ran somewhere specific for reasons unrelated to chance. It needs a decent donor pool of comparable untreated units and enough pre-period history — a couple of years of weekly data is comfortable, a few months usually is not.
Inference works differently and is worth understanding before presenting results. With one treated unit there is no conventional sampling distribution, so significance comes from placebo tests: apply the method to each untreated unit in turn, pretending it was treated, and see how unusual the real gap is against that distribution of fake gaps. If a third of the placebos show a gap as large as yours, you have found nothing, however clean the chart looks. Always check the pre-period fit too, since a synthetic control that tracked poorly before the intervention has no claim to track well after it.
A constrained optimisation over the pre-period, then a subtraction. The constraints are what keep the counterfactual honest.
Ŷ₀ₜ = Σⱼ wⱼ · Yⱼₜ, wⱼ ≥ 0, Σⱼ wⱼ = 1Non-negative weights summing to one forbid extrapolation, so the counterfactual is a real blend of real markets rather than an arithmetic construction.
w* = argmin Σ_{t < T₀} ( Y₁ₜ − Σⱼ wⱼ·Yⱼₜ )²Fitted on the pre-intervention window only. Nothing after T₀ touches the weights, which is why the method cannot be tuned towards a result.
τ̂ₜ = Y₁ₜ − Ŷ₀ₜ for t ≥ T₀A gap per period rather than a single number, so you can see whether the effect builds, holds or decays.
p = rank of |τ̂| among all placebo effects / (J + 1)With one treated unit there is no sampling distribution, so significance comes from how unusual the real gap is against fake ones. A donor pool of 20 gives a minimum p of about 0.05.
A brand launches a sponsorship in one metro area — chosen because the team's flagship store is there, not at random. Weekly revenue per 100,000 population is available for 104 weeks before and 12 weeks after. Thirty-one untreated metros are available as donors.
A 7.1% lift, sustained across twelve weeks, with a placebo p-value of 1/32 = 0.031.
Note what could not have been done here with difference-in-differences: one treated metro, chosen for a reason directly related to its own performance, and no single untreated metro that resembles it. The synthetic blend of four donors tracked it to within 1.4% over two years, which is the evidence that the counterfactual is credible — and that number should always be quoted alongside the effect, because a poor pre-period fit invalidates everything after it. The placebo test is doing the inferential work: applying the same method to all 31 untreated metros produced no gap as large as the real one, which is what a p of 0.031 means here. With a donor pool of 31 that is the smallest p-value available, so a larger pool is the only route to stronger evidence.

Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.


When your advertising efforts are up and sales are increasing, that’s great news, but it doesn't necessarily mean the ads are the reason. Here are three popular tools to measure the lift of a campaign.

Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.