Bell Statistics

Geo testing methods

Measuring advertising by switching it on or off in whole markets — the design decisions, the estimation methods, and what each one assumes.

10 terms

Advertising cannot usually be randomised at the person level. Television is bought by market, people talk to each other, and no cookie survives the journey from a billboard to a purchase. A geo experiment sidesteps all of that by randomising places instead: switch the campaign on or off in whole markets and compare.

The design decisions come first and they bind everything after. The geo unit sets both statistical power and how much media leaks across boundaries, and those pull against each other. Test and control markets are matched rather than split at random, because a few dozen units is far too few for randomisation to balance reliably.

The direction matters as much as the design. A scale-down test measures what your current spend is delivering; a scale-up test measures what more would buy, and it is much less sensitive because the marginal return is smaller than the average. Establishing that current spend works usually has to come first.

The estimation methods — synthetic control, CausalImpact, GeoLift — all build a counterfactual forecast and measure the gap against it. Each entry says what it assumes and how to check whether the assumption held.

Terms in this group

  • CausalImpact

    Fits a time-series model on the pre-period and projects it forward — powerful, easy to run, and easy to run badly.

  • Counterfactual forecast

    The dotted line on every geo chart — a prediction, not an observation, and the whole result rests on it.

  • Designated Market Area

    The 210 US television markets — the standard geo unit because the media buy already respects those boundaries.

  • Geo experiment

    Randomise regions instead of users — the way to test marketing that cannot be hidden from a person.

  • Geo test periods

    Match, measure, then wait — and the cooldown is the phase teams skip and then misread the result.

  • Geo unit

    How finely you cut the map: more units means more power, and more spillover between them.

  • GeoLift

    Open-source geo testing with the power simulation built in — it tells you whether the test can work before you run it.

  • Scale-up vs scale-down test

    Add budget or switch it off — the direction decides which question you get an answer to.

  • Synthetic control

    Build the comparison group instead of finding one — the method for when you have one treated unit.

  • Test and control markets

    Too few markets for randomisation to balance them, so they are matched — and the matching is the whole design.

Geo Testing at Bell Statistics

Geo testing is most of what we do. We design the market split, run the measurement and tell you what the campaign actually caused. See how we work.

Other topics

Back to the glossary