Measuring the true effect of your ads with Geo Testing
Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.
What is the ATT?
The ATT is the average effect among the units that actually received treatment, rather than across the whole population. It is the right quantity when treatment was taken up selectively, and it differs from the ATE precisely because that selection is not random.
ATT
In plain English
The average treatment effect asks what would happen if everyone were treated. The ATT asks something narrower: among the units that actually were treated, how much did it help? In a randomised experiment with full compliance those questions have the same answer, because the treated group is a random sample of the population. As soon as treatment is taken up selectively, they come apart.
The divergence is not a technicality — it is usually the whole story. People who enrol in a loyalty programme, opt into a beta, or respond to a campaign are systematically different from those who do not, and typically differ in ways that relate to the outcome. The effect on that self-selected group can be far larger than the effect would be if you forced the programme on everyone, and the ATT is the quantity that describes it honestly.
Which one you want depends on the decision. If the question is whether to keep running an opt-in programme for the people who choose it, the ATT is correct. If the question is whether to make it mandatory or default-on, the ATE is correct, because it includes the people who would never have opted in and whose response is unknown and often smaller. Answering a rollout question with an ATT is a common and expensive substitution.
Estimating it outside an experiment is where most of the difficulty lies, because it requires constructing what the treated units would have done untreated. Propensity score matching pairs treated units with similar untreated ones. Difference-in-differences uses the untreated group's trend. Both estimate the ATT rather than the ATE by construction, and both rest on the assumption that the comparison group is a valid stand-in — which is exactly what self-selection puts in doubt.
In experimentation the ATT surfaces most often through partial compliance. Assign a feature to half your users, and only some of them ever encounter it. The effect among those who did is an ATT and is not identified by simply comparing them against control, because the users who encountered it are not random. The estimand that is identified in that situation is the LATE, which is a different and narrower thing again.
The formula
The estimand, and the term that makes it different from a naive comparison of the treated against the untreated.
- The estimand
ATT = E[ Y(1) − Y(0) | D = 1 ]Conditioned on having been treated. The second term is the counterfactual for the treated group specifically.
- The naive comparison
E[ Y | D=1 ] − E[ Y | D=0 ] = ATT + ( E[Y(0)|D=1] − E[Y(0)|D=0] )The bracketed term is selection bias: how the treated group would have differed anyway.
- When ATT equals ATE
when D is independent of the potential outcomesTrue under randomisation with full compliance. False whenever uptake is voluntary.
- How it is usually estimated
matching or difference-in-differences on the treated unitsBoth target the ATT by construction — see propensity score matching.
Worked example
A bank offers an opt-in budgeting tool. Adopters save 22% more than non-adopters. To separate the tool's effect from who adopts it, the bank runs a randomised encouragement trial: a random half receive a prompt to try it.
Three defensible numbers — 9.4%, 3.8% and 2.6% — answering three different questions about the same tool.
The naive 22% is the only figure that is simply wrong: most of it is that people who opt into a budgeting tool were already inclined to save. The other three are all correct and answer different decisions. The ATT of 9.4% describes the effect on people who chose the tool, which is the right number for deciding whether to keep supporting it. The LATE of 3.8% describes the effect on people the prompt persuaded — genuinely useful, since that is who a marketing push would reach. The ATE of 2.6% describes what a mandatory rollout would deliver, and it is the smallest because it includes people with no interest who would use it least. Quoting the ATT in a business case for making the tool default-on would overstate the return by a factor of three and a half, which is the ordinary size of this error rather than an extreme one.
Common misconceptions
- “The ATT is what a comparison of treated against untreated units gives you.”
- Only if treatment was randomly assigned. Otherwise that difference equals the ATT plus a selection term — how the treated group would have differed anyway. For a voluntarily adopted feature the selection term usually dominates, which is why naive adopter-versus-non-adopter comparisons are so badly inflated.
- “The ATT is more relevant than the ATE because it describes real users.”
- It depends entirely on the decision. For continuing an opt-in programme, yes. For making something default-on or mandatory, the ATE is the quantity that matters, because it includes the users who never chose it and whose response is usually weaker. Using the ATT for a rollout decision systematically overstates the return.
- “In an A/B test the ATT and ATE are the same, so the distinction never matters.”
- They coincide only with full compliance. When a feature lives on a surface many assigned users never reach, the effect among those who encountered it is not the ATE — and it is not straightforwardly the ATT either, since exposure is self-selected. What is identified there is the LATE.
Frequently asked questions
Should I be estimating the ATT or the ATE?
Match the estimand to the decision. Keeping or improving an opt-in programme is a question about the people who opt in, so the ATT. Making something default-on or mandatory is a question about everyone, so the ATE. The failure to avoid is building a rollout business case on an ATT, which counts only the enthusiasts and typically overstates the return several times over.
How is the ATT estimated without a randomised experiment?
By constructing what the treated units would have done untreated — propensity score matching pairs them with similar untreated units, and difference-in-differences uses an untreated group's trend. Both target the ATT by construction. Both also rest on the comparison group being a valid stand-in, which is precisely the assumption self-selection undermines, so the credibility comes from the design rather than the estimator.
What is an encouragement design and why does it help?
Randomise a prompt to adopt rather than adoption itself, so uptake stays voluntary while assignment is random. That restores a credible comparison and identifies the effect among those the prompt persuaded — the LATE. It is the standard approach for opt-in features where forcing adoption is impossible or undesirable, and it answers the marketing question directly: what happens if we push this.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.