Size matters: How to plan test duration when using CUPED
CUPED is a powerful technique that not only enhances the sensitivity of your tests but also helps in resource optimization by reducing the number of subjects needed for experiments.
What is CUPED?
CUPED is a variance reduction technique that uses each user's pre-experiment behaviour to remove predictable variation from the outcome. The treatment effect is unchanged and the noise around it shrinks, so the same experiment detects smaller effects with the same traffic.
In plain English
Most of the variation in an experiment metric has nothing to do with the experiment. Heavy users were heavy users last month; low-spend accounts were low-spend accounts. That variation is noise as far as the comparison is concerned, and it is what makes the standard error large and the sample size enormous. CUPED — Controlled experiments Using Pre-Existing Data — subtracts the predictable part out, using each user's own history as the predictor.
The mechanism is a regression adjustment. For each user you take the outcome Y during the experiment and a covariate X measured *before* it started — most often the same metric over the preceding weeks — and analyse Y − θX instead of Y, where θ is chosen to minimise variance. Because X was fixed before assignment, it cannot have been affected by the treatment, so subtracting it removes noise without touching the effect. The adjusted estimate is unbiased for exactly the same quantity the unadjusted one estimates.
How much it saves depends entirely on how well the pre-period predicts the experiment period. The variance falls by a factor of 1 − ρ², so a correlation of 0.5 removes 25% of the variance and a correlation of 0.7 removes about half. In practice: revenue and engagement metrics on logged-in users with history often reach 0.5-0.7, which is a 25-50% reduction in required traffic. Conversion on first-time visitors has no pre-period at all and CUPED does nothing. Knowing which of those you have is a five-minute correlation check that should precede any variance-reduction project.
Two conditions have to hold and both are easy to break. The covariate must be measured strictly before assignment — using any data from during the experiment lets the treatment leak into the adjustment and biases the result, which is the one genuinely dangerous mistake here. And θ should be estimated from the pooled data across arms rather than separately per arm, which keeps the adjustment from absorbing part of the effect you are trying to measure.
It changes test planning as well as analysis. Because the variance is lower, the minimum detectable effect for a given duration falls, or the duration for a given MDE falls — and the sample-size calculation has to use the *adjusted* variance or it will demand traffic you no longer need. We work through the planning side in how to plan test duration when using CUPED. New-user experiments, anonymous traffic and cold-start products remain outside its reach, and for those the levers are triggering at exposure and choosing a denser metric.
The formula
One adjustment and one variance identity. Everything about how much CUPED is worth follows from the correlation between the covariate and the outcome.
- The adjusted metric
Y_cuped = Y − θ·( X − X̄ )X is the pre-experiment covariate, X̄ its overall mean. Centring keeps the adjusted metric on the same scale as the original.
- The optimal coefficient
θ = Cov(Y, X) / Var(X)The least-squares slope of Y on X. Estimate it on the pooled data across arms, not within each arm, so the adjustment cannot absorb the treatment effect.
- Variance reduction
Var(Y_cuped) = Var(Y) · (1 − ρ²)ρ = 0.3 removes 9% of the variance; 0.5 removes 25%; 0.7 removes 51%. Sample size falls by the same proportion — see the two-sample t-test calculator.
- Effect on the detectable effect
MDE_cuped = MDE · √(1 − ρ²)At ρ = 0.7 the detectable effect shrinks by 29% for the same traffic, or the duration halves for the same MDE.
Worked example
A subscription product wants to test a new billing page on revenue per user. Unadjusted, the metric has a standard deviation of £42 against a mean of £31, and the required sample for a 3% MDE at 80% power is 143,000 per arm — about nine weeks. Each user's revenue over the four weeks before the test correlates with their revenue during it at ρ = 0.62.
The same experiment, at the same power and the same 3% MDE, runs in five and a half weeks instead of nine. The estimate itself is unchanged.
Three and a half weeks recovered for the cost of joining one pre-period table, which is why this is usually the highest-return change a mature experimentation platform can make. Two details decide whether it holds up. The 81% coverage matters: users without a pre-period get θ·(X − X̄) = 0 and simply carry their unadjusted value, so the realised saving is a little below the headline — plan against the blended figure, not the 38.4%. And the correlation has to be computed on a *past* experiment or a holdout, not on this one, because measuring ρ on live experiment data and then using it to adjust that same data is how the treatment effect quietly leaks into the covariate.
Common misconceptions
- “CUPED makes the effect look bigger.”
- It does not touch the effect estimate, which stays unbiased for the same quantity. What shrinks is the uncertainty around it, so a real effect is more likely to clear significance and the confidence interval is narrower. A technique that moved the point estimate would be a bug, not a feature.
- “We can use any covariate that correlates well with the outcome.”
- Only ones measured strictly before assignment. A covariate collected during the experiment can itself have been affected by the treatment, and subtracting it removes part of the effect along with the noise. This is the one CUPED mistake that produces a wrong answer rather than a weaker one.
- “CUPED helps on any metric.”
- It helps in proportion to ρ², so it does nothing when there is no usable history — first-time visitors, anonymous traffic, newly launched surfaces. Check the correlation between the pre-period and the experiment period before building anything; if it is 0.2, the 4% variance saving is not worth the pipeline.
Frequently asked questions
Which covariate should I use for CUPED?
The same metric measured over a pre-experiment window is the usual first choice and is often the best, because a user's own past behaviour is the strongest available predictor of their future behaviour. Two to four weeks is a common window: long enough to be stable, short enough to still be relevant. If the primary metric is sparse, a denser upstream metric from the same period sometimes correlates better than the metric itself.
Does CUPED work for new users with no history?
Not directly — with no pre-period there is nothing to subtract, and those users contribute their unadjusted values. You can still apply it to the subset that has history and gain proportionally, but for a genuinely new-user experiment the useful levers are different: trigger the analysis at the point of exposure, pick a metric with more events per user, or stratify on whatever is known at signup.
How is CUPED different from just adding controls to a regression?
It is the same idea, and CUPED is essentially a specific, disciplined case of regression adjustment. What the name adds is the constraint that matters: the covariate must be strictly pre-experiment. A general regression invites people to control for variables measured during the test, which can absorb the treatment effect and bias the estimate — the constraint is the whole safety property.
How do I plan sample size when using CUPED?
Compute the pre-period correlation from a past experiment or a holdout, multiply the unadjusted variance by (1 − ρ²), and run the standard sample-size formula on that. Then discount for coverage: users without a usable pre-period get no adjustment, so blend the reduced variance and the unadjusted one in proportion. Planning on the headline reduction and ignoring coverage is the usual way a CUPED-planned test comes up short.
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.