The winner's curse is the systematic overstatement of an effect by the experiment that selected it. Because a result reaches significance partly through luck, the measured effect of a winning variant is biased upward — and the closer it sat to the threshold, the worse the inflation.
An experiment reports a 6% lift and clears significance. It ships, and the following quarter the aggregate metrics show something closer to 3%. Nobody made an error. The measured effect of a variant that was selected for being significant is systematically larger than its true effect, because reaching the threshold required both a real effect and a favourable draw of noise — and the noise is not repeated when the change goes live.
The mechanism is selection on a noisy estimate. Suppose the true effect is 3% and the experiment's precision is such that observed results scatter by a couple of points either side. Runs where noise pushed the estimate up cross the significance line; runs where it pushed down do not. The winners are therefore drawn disproportionately from the upper half of that scatter, and their average overstates the truth. This is regression to the mean, arriving in the specific form that matters for experimentation.
The size of the inflation depends almost entirely on how much power the test had. A well-powered experiment that clears significance comfortably is barely affected — the observed effect is large relative to the noise, so selection changes little. A marginal result at p = 0.048 from an underpowered test can be inflated by a factor of two or more. The uncomfortable implication is that the most exciting results, from the smallest tests, are the least trustworthy in magnitude.
Multi-arm tests compound it. Taking the best of five variants selects the maximum of five noisy estimates, and the maximum of a set is upward-biased even when every arm has an identical true effect. That is the same phenomenon operating twice — once through the significance threshold, once through picking the leader — which is why the winner of an A/B/n test should be treated with particular caution.
The practical responses are modest and effective. Plan against the lower end of the confidence interval rather than the point estimate. Discount marginal results more heavily than comfortable ones, since the inflation scales with how close the call was. And where a shipped change matters enough, hold a global holdout to measure what actually accumulated — realisation rates around a third are common, and this is one of the main reasons why.
The bias is a conditional expectation, and its size is governed by how far the significance threshold sits from the true effect in standard-error units.
E[ effect̂ | significant ] > true effectConditioning on having crossed the threshold selects the favourable half of the noise.
inflation grows as power fallsAt 80% power the bias is modest; at 20% power a significant estimate can be double the truth.
E[ max of k estimates ] > true effect, even with identical armsSelecting the leader is a second layer of the same selection — see the ANOVA calculator.
plan against the lower bound of the CI, not the point estimateCheap, requires no modelling, and errs in the direction that does not embarrass anyone.
A simulation gives every experiment in a programme a true effect of exactly 3%. Experiments are run at three power levels, and only those reaching significance are shipped. The measured effect of the shipped ones is compared against the truth.
The same 3% true effect is measured at 3.2%, 4.4% or 6.7% depending only on how well powered the test was.
Nothing varies across those rows except statistical power, and the reported effect more than doubles. That is the whole argument for treating underpowered significant results as unreliable in magnitude even when the direction is right — the 20% power row is not measuring a different effect, it is measuring the same one through a filter that only lets the lucky draws through. The last row adds the multi-arm layer and takes the inflation to 170%. Two consequences worth carrying. Forecasts built on the point estimate of a marginal win will be wrong by a factor that is predictable in direction and not in size, so the lower bound of the interval is the safer planning number. And a programme whose experiments are mostly underpowered will report a stream of large wins that never appear in the aggregate — which is a measurement artefact rather than an execution failure, and only a holdout distinguishes the two.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.