What is a global holdout?

Individual experiments answer individual questions. A global holdout answers a different and harder one: across everything this team shipped last quarter, what actually changed? It works by permanently withholding all releases from a small randomly chosen slice of users — typically 1 to 5% — and comparing them against the population receiving normal development.

The reason to bother is that summing experiment results reliably overstates reality, usually by a large factor, and every mechanism that causes this is invisible from inside the individual tests. Novelty effects inflate short-horizon readings. Winner's curse means a result that barely cleared significance is more likely to have been lucky than to be at its estimated size. Interactions mean two changes that each worked can cancel when shipped together. And proxy metrics transfer to real outcomes at a fraction of the assumed rate. Each experiment can be correctly run and correctly analysed and the total still not appear.

Organisations that do this consistently report the same shape of finding: the cumulative effect measured against a global holdout is materially smaller than the sum of the wins, often by half or more. That is not a sign the programme is failing. It is a correction to a systematic overstatement that no other measurement can catch, and knowing the real number is what makes forecasting and investment decisions honest.

Two properties make it expensive to run. It requires infrastructure discipline — every experiment and every release must respect the holdout, and a single team shipping around it silently invalidates the comparison for everyone. And it has weak statistical power, because precision is set by the small arm: a 2% holdout of a million users gives roughly the precision of a 20,000-user experiment, which resolves only fairly large cumulative effects. That is usually acceptable, since the question is about aggregate movement rather than a single change.

Distinguish it from a holdout group, which withholds one specific change and is tied to it. The global version is not attached to anything: it is the counterfactual for the whole development effort. The practical cost is that users in it drift progressively further from the current product, which raises a genuine question about how long anyone can reasonably be held there — most teams rotate the membership periodically, accepting a small loss of comparability to avoid stranding a group on a year-old experience.

A consumer app maintains a 3% global holdout for a full year. Over that period the team shipped 34 experiments that reached significance on their primary metric, weekly sessions per user, with reported lifts summing to 22%. At year end the holdout is compared against the rest.

A 34% realisation rate sounds like a failure and is close to what well-run programmes report, which is the useful calibration here. The individual experiments were not wrong — each measured a real short-horizon effect on the population it was run on. What the sum ignores is that novelty decay removed part of several, that changes shipped in the same quarter competed for the same attention, and that a handful of the 34 were borderline results that were probably overstated at the point of shipping. The right response is to keep the realisation rate as a standing multiplier on forecasts rather than to distrust the experiments. Two cautions on reading this table. The interval spans 4.9% to 9.9%, so the realisation rate is somewhere between 22% and 45% rather than exactly a third. And the absence of a retention difference is reassuring but weakly evidenced: at 72,000 holdout users only a fairly large retention effect would have been detectable.