What is stratified randomization?

Plain randomization balances everything in expectation and nothing in particular on any given run. Split 4,000 users evenly and you can easily end up with 8% more mobile traffic in one arm — not because anything is broken, but because that is what random variation does at that scale. If mobile users behave differently, that imbalance sits inside your result and cannot be distinguished from the treatment effect.

Stratified randomization removes the possibility for characteristics you name in advance. Divide users into strata — mobile and desktop, or new and returning, or by country — and run a separate balanced assignment within each. Both arms then contain exactly the same proportion of each stratum by construction. Blocking is the same idea under a different name, more common in clinical trials and in geo experiments, where markets are grouped into blocks of similar size before assignment.

The benefit shows up in two ways. Balance on the stratifying variable is guaranteed rather than probable, which removes a source of confounding that plain randomisation only makes unlikely. And when the variable predicts the outcome, variance falls, because the between-stratum differences no longer contribute noise to the comparison — the same logic that makes a paired t-test more sensitive than an unpaired one, applied at group level rather than individual level.

How much it buys depends almost entirely on scale, and this is the part most often misstated. At a few thousand users, imbalance is a genuine risk and stratifying is valuable. At a million users per arm, plain randomisation already balances every characteristic to within a fraction of a per cent, and the variance reduction from stratifying is typically under 2% — real, but not worth a pipeline. Its home is small experiments, geo tests with a few dozen markets, and any situation where one stratum is both small and very different from the rest.

There is a cheaper alternative for large experiments that captures most of the same benefit: analyse with regression adjustment rather than stratifying at assignment time. Including the covariate in the model at analysis removes its contribution to the variance without needing anything special in the assignment layer, and it can be added retrospectively. Stratification's advantage over that is guaranteed balance, which matters when the sample is small enough for imbalance to be plausible.

A B2B product tests a pricing page change with 3,200 accounts, which is all the traffic available in a month. Account size varies enormously — 74% are small, 21% mid-market, 5% enterprise — and enterprise accounts convert at eight times the small-account rate. The team compares plain and stratified assignment by simulation before choosing.

Both numbers matter and the second is the more important one. A 31% variance reduction is a useful efficiency gain, equivalent to roughly 45% more traffic. But the 18-in-100 chance of a badly skewed enterprise split is the real risk: with 160 enterprise accounts converting at eight times the base rate, an arm receiving 20% more of them gains an apparent lift that has nothing to do with the pricing page. That is a confounded result that looks exactly like a successful test, and no amount of correct analysis afterwards can separate it from the treatment. This is the regime stratification is for. Worth stating the contrast plainly: at 3.2 million accounts rather than 3,200, the same simulation would show essentially zero chance of meaningful imbalance and a variance reduction of well under 2%, and the sensible answer would be plain randomisation with the covariate handled at analysis.