Variance reduction and efficiency

Getting a usable answer from less traffic: removing predictable noise, taming the outliers that dominate a mean, and analysing ratios correctly.

6 terms

The sample an experiment needs scales with the square of how noisy its metric is, which means noise is the most expensive thing in experimentation. Halve the variance and you halve the traffic, or detect an effect two-thirds the size in the same time. The techniques in this group all do that, and none of them changes what is being estimated — the effect stays the same, the uncertainty around it shrinks.

The largest single win is usually CUPED, which uses each user's own pre-experiment behaviour to subtract out the part of their outcome that was predictable before the test began. Where users have history it removes 20 to 50 per cent of the variance for the cost of one join. Where they do not — new visitors, anonymous traffic — it does nothing at all, and knowing which case you are in is a five-minute correlation check.

The other reliable source of savings is the tail. Revenue metrics are routinely dominated by a handful of enormous customers, and one landing in the treatment arm can move the result more than the treatment did. Winsorization caps them at a threshold fixed in advance, which is the whole safeguard.

The rest of the group is about getting the arithmetic right rather than reducing it. The delta method supplies the correct variance for a ratio whose denominator is random, and stratified randomization balances known characteristics across arms before the test starts rather than adjusting for them afterwards.