Variance is the average squared distance between each observation and the mean. It is the squared version of standard deviation, harder to interpret directly, and the form that adds cleanly across independent sources — which is why it appears inside almost every statistical formula.
σ²Variance is the average of the squared distances between observations and their mean. Squaring does two things: it makes negative and positive deviations count equally, and it weights large deviations far more heavily than small ones. The cost is that the units are squared — variance in pounds is measured in pounds-squared, which nobody can picture — which is why standard deviation exists as its square root and is what gets reported.
The reason variance survives despite being unreadable is that it is additive. For independent quantities, the variance of a sum is the sum of the variances, which is not true of standard deviations. That single property is what makes derivations tractable: the variance of a difference between two independent group means is the sum of their two variances, which is where the standard error of every two-sample test comes from. Try to do that arithmetic with standard deviations and it does not work.
It is also the quantity every sample-size formula is denominated in. The number of observations required scales directly with variance, not with standard deviation, which is a distinction with a large practical consequence: halving the standard deviation quarters the required sample. That is why variance reduction is worth so much more than it sounds. A technique that removes 40% of the variance removes 40% of the required traffic, and one that halves the standard deviation removes 75% of it.
In experimentation the useful mental model is that variance comes from several sources and only some of them are informative. Some of it is the treatment effect you are looking for. Much more of it is stable differences between users — heavy users versus light ones — which is predictable from history and therefore removable with a pre-period covariate. Some is day-of-week and seasonal structure, removable by design. And some is genuinely irreducible noise. Variance reduction is the practice of identifying which parts are predictable and subtracting them.
One warning for skewed data. Because deviations are squared, a single extreme observation contributes enormously — a value ten standard deviations out adds a hundred times what a one-sigma value adds. On a revenue metric with a long tail, the variance is often dominated by a handful of orders, which makes it unstable between samples and makes any interval built on it fragile. Capping or winsorising is the standard response, and it has to be decided before results are seen.
The definition, the additivity property that makes it useful, and the two applications that matter most in practice.
s² = Σ(xᵢ − x̄)² / (n − 1)n − 1 rather than n corrects the bias from measuring deviations against the sample's own mean.
Var(X + Y) = Var(X) + Var(Y) for independent X, YThe property standard deviations lack, and the reason variance appears in every derivation. Add a covariance term when they are not independent.
Var(x̄₁ − x̄₂) = σ₁²/n₁ + σ₂²/n₂Straight from additivity, and the basis of every two-sample test — see the two-sample t-test calculator.
n = 2·(z₁₋α/₂ + z₁₋β)² · σ² / δ²Directly proportional. Removing 40% of the variance removes 40% of the traffic needed, which is the entire case for CUPED.
A revenue-per-user metric has a variance of 8,281 (a standard deviation of £91). An analyst decomposes it to see what could be removed, using a pre-experiment window and a day-of-week decomposition.
CUPED on the pre-period covariate removes the 43%; winsorising at the 99th percentile removes most of the 29%. Together the required sample falls from 184,000 to about 60,000 per arm.
Two thirds of this metric's variance had nothing to do with the experiment. The stable user differences are the largest single block and the easiest to remove, because a user's own history predicts it — that is exactly what CUPED subtracts. The top 1% of orders is the second block and the more delicate one: winsorising removes it, and it also changes what you are measuring, since a treatment that genuinely produces very large orders would now be partly invisible. That is an acceptable trade for most conversion-oriented tests and a bad one for a test aimed at high-value customers, which is why the decision belongs in the design document rather than in the analysis. The threefold reduction in sample is what a fortnight instead of six weeks looks like.
In A/B testing, you often have to balance statistical power and how long the test takes. Learn how Allocation, Effect Size, CUPED & Binarization can help you.
CUPED is a powerful technique that not only enhances the sensitivity of your tests but also helps in resource optimization by reducing the number of subjects needed for experiments.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.