A geo experiment randomises whole geographic regions rather than individual users, running marketing in some and withholding it in others. It recovers a genuine randomised comparison for anything that cannot be hidden from an individual person.
A/B testing needs to show different things to different people, which rules it out for most marketing. You cannot show a television advertisement to half a household. You cannot hide a billboard, a radio spot, or a price change from a randomly chosen cookie. And even where you technically can — social and display — a user seen on three devices and two browsers is not one unit. A geo experiment sidesteps all of this by moving the randomisation up a level: regions are assigned to treatment or control, spend is set accordingly, and outcomes are compared per head of population.
What you buy is a real causal estimate for channels that otherwise have none. Because assignment is random, the treatment and control regions are comparable in expectation on everything — including local demand, competitive intensity and the seasonal patterns that make observational marketing analysis so treacherous. That makes a geo test the most direct available measurement of incrementality: the difference between regions is what the advertising caused, not what it was present for.
What you pay is statistical power. A country has perhaps 50 to 210 usable regions, so the effective sample size is the number of regions rather than the number of people, and regions vary enormously in size and character. This is why geo tests detect double-digit lifts comfortably and struggle below about 5%, and why the design work matters more than in a user-level test: stratify regions into blocks of similar size and baseline before randomising, or match pairs and randomise within them. A test that assigns the three largest metros to treatment by chance has burnt most of its precision before collecting a single data point.
The analysis is rarely a simple difference in means. Regions differ in level, so the standard approaches are difference-in-differences against a pre-period, or a synthetic control per treated region — and the second is preferred when regions were not randomised, which happens more often than anyone admits. Standard errors must be clustered at the region, since outcomes within a region across weeks are highly correlated, and ignoring that produces intervals several times too narrow.
Three practical constraints decide whether a geo test is feasible. Spillover: regions must be far enough apart that treated advertising does not reach control areas, which rules out adjacent metros for broadcast media and makes online targeting accuracy a real concern. Duration: four to eight weeks is typical, long enough for effects to accumulate and short enough that the regions do not drift apart for unrelated reasons. And a clean pre-period of several months, because almost every analysis method leans on it. We cover the design decisions in how to tackle your marketing challenges with geo tests.
The estimator is ordinary; the sample size arithmetic is where geo tests differ from user-level ones, because n is the number of regions.
Yᵢ = revenueᵢ / populationᵢNormalising by population is what makes regions of wildly different size comparable. Weighting by population afterwards recovers the aggregate effect.
k = 2·( z₁₋α/₂ + z₁₋β )² · σ²_region / δ²σ² is the variance ACROSS regions, not across people. This is why a country with 60 usable regions supports a 10% test and not a 3% one — see the two-sample t-test calculator.
τ̂ = ( Ȳ_t,post − Ȳ_t,pre ) − ( Ȳ_c,post − Ȳ_c,pre )Removes fixed differences between regions and anything that affected all of them over the window.
iROAS = ( τ̂ × population_treated ) / incremental spendThe number the budget decision needs. Use the interval as well as the point estimate — geo tests are noisy enough that the two often imply different decisions.
An advertiser wants to know whether a connected-TV campaign is worth £1.2m a quarter. There are 68 usable metro areas. They stratify into 34 pairs matched on population and baseline revenue per head, randomise one of each pair into treatment, and run for six weeks with twelve weeks of pre-period.
Incremental revenue of about £2.76m against £600,000 of spend — an iROAS of 4.6, with the interval running from 1.4 to 7.7.
The point estimate says the campaign returns £4.60 per pound, and the interval says it returns somewhere between £1.40 and £7.70. If the business needs 2.0 to justify the channel, the honest summary is that it probably clears the bar but the test could not confirm it — and that width is the characteristic cost of geo testing, where 68 regions is the sample size no matter how many millions of people they contain. Two design choices did the heavy lifting: matched pairs prevented a randomisation that put the big metros on one side, and clustering at the region kept the interval honest. Analysing the same data at week level without clustering would have produced an interval perhaps a third as wide and a false sense of precision.
Our “Geo Testing: Unlocking True Incrementality” webinar explored how teams can measure real-world impact when A/B testing isn’t possible - using geographic experiments and synthetic controls to reveal true incremental lift in marketing and product initiatives.
Traditional methods often fall short in measuring the true impact of marketing strategies. Here's how Geo Tests can resolve common marketing challenges.
Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.
Region selection, matching and power are where geo tests are won or lost, long before the analysis — see Geo Testing