Size matters: How to plan test duration when using CUPED
CUPED is a powerful technique that not only enhances the sensitivity of your tests but also helps in resource optimization by reducing the number of subjects needed for experiments.
What is regression adjustment?
Regression adjustment reduces the noise in an experiment by including pre-treatment covariates in the analysis model. The treatment effect is estimated after accounting for what those covariates already explain, which narrows the interval without changing what is being estimated.
In plain English
Most of the variation in an experiment's outcome has nothing to do with the experiment. Users differ in tenure, device, country, and how much they were already spending, and all of that difference is noise as far as the treatment comparison is concerned. Regression adjustment removes the part of it that you can predict: instead of comparing raw group means, fit a model with the treatment indicator plus the covariates you knew before assignment, and read the treatment coefficient.
The estimate does not change and the uncertainty around it shrinks. That is the property worth internalising, and it holds because assignment was random — the covariates are balanced across arms in expectation, so removing their contribution cannot bias the treatment effect, only sharpen it. How much it sharpens depends entirely on how much the covariates explain: variance falls by roughly the model's R², so covariates explaining 40% of the outcome cut the required sample by about 40%.
CUPED is this method with one specific covariate — the same metric measured before the experiment — and that is usually the most predictive covariate available, which is why it gets its own name and its own literature. Regression adjustment is the general form: use the pre-period metric and tenure and device and country together, and take whatever additional variance they collectively explain. The gain over CUPED alone is often modest, since a user's own past behaviour tends to dominate, but it is free once the model exists.
The one rule that must not be broken is that every covariate is measured strictly before assignment. A variable recorded during the experiment may itself have been affected by the treatment, and adjusting for it removes part of the effect you are trying to measure — turning a variance reduction into a bias. This is the same constraint CUPED carries and for the same reason, and it is the mistake that produces a wrong answer rather than merely a weaker one. Number of sessions during the test is the tempting example: highly predictive, and completely disqualified.
Compared with stratified randomization this is the analysis-side answer to the same problem, and for large experiments it is usually the better trade. It requires no change to the assignment layer, can be applied retrospectively to an experiment already run, and handles continuous covariates naturally rather than forcing them into buckets. What it does not provide is guaranteed balance, which is what stratification is for when the sample is small enough for a bad split to be plausible.
The formula
One model, and one identity that says what it buys. The interaction form is worth knowing because it is what makes the estimator robust when the covariate relationship differs between arms.
- The model
Y = β₀ + β₁·treated + β₂·X + εβ₁ is the treatment effect, estimated after X has absorbed what it explains. X must be pre-treatment.
- What it buys
Var_adjusted ≈ Var_unadjusted · ( 1 − R² )R² is the share of outcome variance the covariates explain. 40% explained means about 40% less required traffic.
- The CUPED special case
X = the same metric, measured pre-experimentThen R² = ρ² and this reduces to CUPED's (1 − ρ²) — see the sample size calculator.
- The interacted (Lin) estimator
Y = β₀ + β₁·treated + β₂·X + β₃·( treated × X )Centre X first. Guarantees the adjustment cannot hurt precision even if the covariate relationship differs by arm.
Worked example
A subscription product runs a test on monthly revenue per user with 62,000 users per arm. The unadjusted comparison is inconclusive. The team refits with four pre-treatment covariates they already hold: pre-period revenue, tenure in months, plan tier, and device class.
The point estimate barely moves — £0.41 to £0.40 — and the interval narrows by 30%, taking the result from inconclusive to significant.
The stability of the point estimate is the reassuring part and the thing to check first: adjustment should sharpen an estimate, not move it. A large shift would suggest either that randomisation failed or that one of the covariates is not genuinely pre-treatment, and either is worth investigating before believing the adjusted result. The incremental value of the three extra covariates is worth noting too — pre-period revenue alone gets R² to 0.44, and tenure, plan and device together add only 0.08. That is the usual pattern, and it is why CUPED gets its own name: a user's own past behaviour dominates everything else you know about them. The temptation this example should not encourage is adding whatever else is available. Sessions during the test period would push R² well above 0.7 and would be disqualified, because the treatment plausibly changed it.
Common misconceptions
- “Adjusting for covariates is a way of fixing an unbalanced experiment.”
- It reduces variance in a properly randomised experiment; it does not repair a broken one. If assignment was compromised — a sample ratio mismatch, or a filter applied after assignment — adjustment cannot recover the comparison, because the arms differ on things you did not measure as well as things you did.
- “More covariates always means a better adjustment.”
- Only covariates that predict the outcome help, and each one costs a degree of freedom. Beyond the few strong predictors the gain flattens quickly, and with many weak covariates in a small sample the model starts fitting noise. In practice the pre-period metric does most of the work and a handful of others add a little.
- “Any variable in the dataset can be used as a covariate.”
- Only ones measured strictly before assignment. A variable recorded during the experiment may have been affected by the treatment, and adjusting for it removes part of the effect along with the noise. This turns a variance reduction into a bias, and it is the one mistake here that yields a confidently wrong answer.
Frequently asked questions
How does regression adjustment differ from CUPED?
CUPED is regression adjustment using one specific covariate: the same metric measured before the experiment. That covariate is usually the most predictive one available, which is why it has its own name and tooling. Regression adjustment is the general case, letting you add tenure, device, plan and anything else known in advance. The extra covariates typically add much less than the pre-period metric already did.
Which covariates should I include?
Anything measured before assignment that predicts the outcome, starting with the pre-period value of the metric itself. Check the incremental R² as you add each one and stop when it flattens — usually after three or four. The absolute rule is the timing: if a variable could have been influenced by the treatment, it is disqualified regardless of how predictive it looks.
What is the interacted or Lin estimator and should I use it?
It adds treatment-by-covariate interaction terms with the covariates centred first, which guarantees the adjustment never reduces precision relative to the unadjusted comparison even when the covariate relationship differs between arms. Without the interactions there are edge cases where adjustment can hurt. The cost is a few extra parameters, so with large samples and a handful of covariates it is close to free and worth defaulting to.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.