What is the average treatment effect?

The average treatment effect is the difference between two hypothetical worlds: everyone in the population treated, and nobody treated. Written in potential outcomes it is the mean of Y(1) minus Y(0) across all units. Neither world is observed, and the value is nonetheless estimable — because randomisation makes the treated group's average a valid estimate of the first and the control group's a valid estimate of the second.

This is what a properly run A/B test delivers, and it is worth appreciating how much randomisation is doing. Without it, the treated and untreated groups differ in ways you cannot measure, so their difference confounds the effect with those differences. With it, the two groups are alike in expectation, so a simple difference in means is an unbiased estimate of the ATE with no modelling required at all.

The word doing the work is average. If a change helps mobile users by 8% and hurts desktop users by 3%, the ATE might be 2% — a number that describes neither group and would be the wrong basis for a rollout decision that could be made per platform. Averaging is not a flaw; it answers exactly the question "what happens if we ship this to everyone". It becomes misleading only when treated as a description of a typical user, which it is not.

It is one of several estimands and the distinctions matter when compliance is imperfect. The ATT is the effect among those who actually received treatment, which differs from the ATE when who gets treated is not random. The LATE is the effect among those whose treatment status responds to the assignment. In a clean randomised experiment with full compliance all three coincide, which is why the distinction rarely surfaces in product work and becomes essential in observational settings.

Two practical cautions. The ATE is specific to the population tested — an effect estimated on existing users says little about prospective ones, and one estimated during a promotion says little about a quiet week. And the answer to whether the average conceals variation is a question about heterogeneous treatment effects, which is worth asking whenever a rollout could be targeted rather than universal.

A subscription product tests a simplified onboarding flow across its whole user base. The overall result is reported, then broken down by whether the user arrived from paid or organic acquisition — a split decided in advance.

The ATE of +1.8 points is correct and is the right answer to "what happens if we ship this to everyone". It is the wrong answer to "should we ship this", because a decision that could be made per segment has better options available than the average implies. Paid-acquisition users, who arrive with less context, benefit substantially from a simpler flow; organic users, who arrive already informed, are slowed by it. Shipping to paid only would deliver 4.1 points on 62% of the base rather than 1.8 on all of it. Two cautions before acting on that. The split was decided in advance, which is what makes the interaction test at p = 0.003 meaningful rather than the product of searching many segmentations. And targeting by acquisition source means maintaining two flows, which has a real engineering cost that the statistics do not capture.