Size matters: How to plan test duration when using CUPED
CUPED is a powerful technique that not only enhances the sensitivity of your tests but also helps in resource optimization by reducing the number of subjects needed for experiments.
How do you calculate sample size?
Sample size is how many observations an experiment needs to detect an effect of a chosen size with a chosen reliability. It is derived from four inputs — baseline, minimum detectable effect, significance level and power — and it must be fixed before the test starts.
n
In plain English
Sample size is not a thing you choose; it is a thing you compute. Four numbers go in — the baseline rate of your metric, the smallest effect worth detecting, the significance level and the statistical power you want — and the required n comes out. Three of those four are conventions you will barely think about. The one that does the work is the minimum detectable effect, and it is a business judgement rather than a statistical one.
The relationship that governs everything is the square law. The effect size appears squared in the denominator, so halving the effect you want to detect multiplies the sample by four. This is why a test for a 10% lift is comfortable and a test for a 2% lift on the same metric needs twenty-five times the traffic — and why teams so often discover that the improvement they care about is not measurable at their scale. Better to learn that from a calculation than from six weeks of running.
Fixing the number in advance is the part that makes the whole procedure honest. A test whose duration is decided by watching the p-value has an error rate several times its nominal one, because every look is another chance to cross the threshold. If you genuinely need to monitor continuously — and there are good reasons to — use sequential testing, which is designed for it and charges the error budget correctly as it goes. What you cannot do is run a fixed-horizon test and treat it as if it were sequential.
Once you have the number, round the duration up to whole weeks. Behaviour differs systematically between weekdays and weekends, and a test that stops mid-week contains an unbalanced mix of days. That does not bias the comparison, since both arms see the same days, but it does add variance, and two full weeks also gives novelty effects room to decay so you can compare week one against week two.
The most useful move when the number comes back too large is not to compromise on power but to reduce variance. CUPED using a pre-period covariate commonly cuts the requirement by 20-50% on metrics with a strong pre-period correlate. Triggering the analysis at the point of exposure rather than at assignment often does more. Choosing a less volatile primary metric, or winsorising an outlier-heavy revenue measure, both work on the same lever. All of these buy sensitivity without buying traffic, which is the only free lunch in the subject.
The formula
One formula per outcome type, both the same shape: a constant set by alpha and power, times the variance, divided by the squared effect.
- Two proportions, per arm
n = ( z₁₋α/₂ + z₁₋β )² · ( p₁(1−p₁) + p₂(1−p₂) ) / (p₁ − p₂)²The conversion-rate case. At α = 0.05 and 80% power the leading constant is (1.960 + 0.842)² = 7.85.
- Two means, per arm
n = 2·( z₁₋α/₂ + z₁₋β )² · σ² / δ²For revenue or duration. σ² is the variance of the outcome, which is why variance reduction is worth as much as extra traffic — see the two-sample t-test calculator.
- The square law
n ∝ 1 / δ²Halve the detectable effect, quadruple the sample. This single relationship explains most of the surprise in sample-size conversations.
- Cost of more power
n₉₀ / n₈₀ = ((1.960 + 1.282) / (1.960 + 0.842))² = 1.34Going from 80% to 90% power costs 34% more sample; 95% costs 66% more. Usually cheaper than the cost of a false negative on an irreversible decision.
Worked example
A subscription product converts trials at 11.5%. The team wants to detect a 4% relative improvement — 11.5% to 11.96% — at the conventional 5% two-sided level and 80% power. They start 9,000 trials a week and would split evenly between two arms.
About 13.7 weeks per arm — a fourteen-week test. At a 6% MDE it drops to about six weeks; at 8%, to three and a half.
Fourteen weeks is not an experiment, it is a quarter, and over that horizon the product, the traffic mix and the season will all change underneath it. The square law is what makes the alternatives so unequal: relaxing the MDE from 4% to 6% is a 1.5× change in the effect and a 2.25× change in the duration, which is the difference between a test you can run and one you cannot. So the real question is whether a 4% lift is genuinely the smallest one worth shipping, or whether that number was chosen because it sounded modest. If it is genuine, the answer is variance reduction — a pre-period covariate on a subscription funnel typically has a strong correlation and could bring fourteen weeks closer to eight — not a shorter test at 40% power, which would answer nothing.
Common misconceptions
- “We will run the test until it reaches significance.”
- That turns a 5% false-positive rate into roughly 25% over a fortnight of daily checks, because every look is another chance to cross the threshold. It also biases the winning effect size upward, since you stop precisely when noise is working in your favour. Fix the horizon in advance, or use a sequential design built for continuous monitoring.
- “More traffic is always the answer when the test is too small.”
- It is the most expensive answer. Variance reduction, triggering at the exposure point, and choosing a less volatile metric all raise sensitivity without a single extra user, and the first two are usually worth more than doubling the sample. Reach for traffic once you have taken the free options.
- “We can decide the minimum detectable effect after seeing the result.”
- Then the test was powered for whatever happened to occur, which is not a design. The MDE has to come from what improvement would justify building and maintaining the change — a business question — and it has to be answered before the data can influence it.
Frequently asked questions
Is the calculated sample size per arm or in total?
Per arm, in the standard formulas — so a two-arm test needs roughly twice the figure in total, and a four-arm test four times it. This is the most common arithmetic slip in test planning and it makes experiments run at half the intended power. If a calculator does not say which it is reporting, assume per arm and check against a second tool.
Can I run an unequal traffic split, like 90/10?
You can, and it is sometimes the right call for a risky change, but it costs sensitivity. An even split maximises power for a fixed total sample, and a 90/10 split needs roughly 2.8 times the total traffic to match it, because the smaller arm becomes the binding constraint. Use unequal splits to limit exposure to risk, not to run a test faster.
Why does my revenue metric need so much more sample than conversion?
Because revenue is far more variable. Conversion is a zero-or-one outcome with a variance capped by the rate itself, while revenue per user is skewed by a small number of very large purchases, and the sample size scales directly with variance. Winsorising or capping the extreme values, or analysing the log, will usually cut the requirement dramatically — and the choice should be made before you look at the results.
What if I do not have enough traffic for any reasonable test?
Then say so explicitly and change what you test rather than pretending. At low traffic, only large effects are detectable, so the strategy is to test bold changes rather than incremental refinements, and to accept that a flat result rules out nothing smaller than your actual detectable effect. Moving to a metric earlier in the funnel, which has a higher base rate and more events, is often the single most effective adjustment.
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.