In plain English
An experiment ends when the decision is made, and for most changes that is fine. For some it is not: effects on retention, churn, habit formation and lifetime value take months to appear, and a two-week test cannot see them. A holdout group is the answer — a slice of users kept on the old experience after the change has shipped to everyone else, so the comparison continues.
The distinction from a control group is purpose rather than mechanism. Both are randomly assigned and both receive the old experience. A control exists to decide an experiment and ends with it; a holdout exists to keep measuring one after the decision, often for a quarter or longer. In practice a holdout is frequently just the control arm of a concluded test that was never released, which is the cheapest way to create one.
What it buys is the ability to detect three things an experiment structurally cannot. Effects that appear late — a change that lifts conversion immediately and increases churn by month three. Effects that decay — the novelty effect, where an early lift fades once the change stops being new. And accumulated effects, where a year of individually modest improvements is measured in aggregate rather than by summing test results that each carried their own optimism.
The cost is real and should be stated honestly. A holdout deliberately withholds an improvement from some users, which for a genuinely good change means leaving value on the table for the length of the hold. It also complicates the codebase, since the old path must be maintained alongside the new one, and that maintenance burden is usually what kills long holdouts rather than any statistical argument. Both costs scale with duration and with the size of the group.
Sizing follows from that trade. The holdout only needs to be large enough to detect the effect you care about, and precision is limited by the smaller arm — so a 5% holdout against 95% treated has roughly the power of a 5% versus 5% comparison, not of the full population. For most consumer products, 1 to 5% held for a quarter is the usual range. A permanent version of this is the global holdout, which receives nothing at all rather than being tied to one change.
The formula
The comparison is an ordinary difference. What decides whether a holdout is worth running is the power arithmetic, which is governed by the smaller arm.
- The comparison
effect = E[ Y | shipped ] − E[ Y | holdout ]Same estimator as an experiment, run over a longer horizon and after the decision.
- Power is set by the small arm
SE ≈ σ · √( 1/n_holdout + 1/n_treated )With n_treated ≫ n_holdout the second term vanishes, so precision is effectively that of the holdout alone.
- What a 5% holdout costs in power
n_effective ≈ n_holdout5% of a million users gives the precision of a 50,000-user arm, not of a million — see the sample size calculator.
- The value withheld
cost = holdout share × effect × durationReal, and proportional to how good the change is. Small holdouts held briefly keep it modest.
Worked example
A subscription product ships a recommendation change that tested at +3.2% on weekly engagement over two weeks. A 4% holdout is kept for six months to check whether the gain persists and what it does to retention, which the original test could not measure.
- Original experiment
- +3.2% weekly engagement, p = 0.004
- Holdout size
- 4% of 1.1 million users = 44,000
- Engagement gap, month 1
- +3.0%
- Engagement gap, month 6
- +1.1%
- 6-month retention, shipped
- 68.4%
- 6-month retention, holdout
- 69.1%
The engagement gain decays to about a third of its original size, and retention is 0.7 points lower in the shipped group.
Neither of these was visible in the original two-week experiment, and both change the decision. The engagement decay is a partial novelty effect — a real but smaller durable gain than the test suggested, which by itself would be acceptable. The retention difference is the serious finding: 0.7 percentage points on 1.1 million subscribers is a large number, and it points at the recommendation change surfacing content that drives short-term engagement at the cost of long-term satisfaction. Two cautions about reading this. At 44,000 holdout users the retention difference needs its own confidence interval before anyone acts — a 0.7-point gap at that size is around the edge of detectability, and treating it as established would be overreaching. And the decision is not automatically to revert; it is to investigate which part of the change drives the retention effect, since the engagement gain may be worth keeping if that part can be isolated.
Common misconceptions
- דA holdout is the same as a control group.”
- Mechanically similar, different in purpose and lifespan. A control decides an experiment and ends with it; a holdout continues after the decision to measure effects over a longer horizon. Most holdouts begin as the control arm of a finished test that was simply never released.
- דA small holdout is enough because the treated group is huge.”
- Precision is governed by the smaller arm. A 2% holdout against 98% treated has approximately the power of a 2% versus 2% comparison, because the standard error is dominated by whichever side has fewer users. Size the holdout for the effect you want to detect rather than as a percentage that sounds cautious.
- דHoldouts are free because you are only withholding from a small group.”
- They withhold a genuine improvement from those users for the duration, and they require maintaining the old code path alongside the new one. The engineering cost is usually what ends long holdouts in practice. Both costs are worth paying for changes with plausible long-horizon effects and not for cosmetic ones.
