Cluster randomization assigns whole groups — accounts, households, markets or network communities — to a variant rather than assigning individuals one at a time. It is the standard remedy for interference: the situation where treating one person changes another person's outcome and breaks the comparison.
Individual randomisation rests on an assumption that one person's assignment does not affect another person's outcome. It fails whenever people interact: colleagues in a shared workspace, friends on a social network, buyers and sellers in a marketplace, households sharing a television. Cluster randomization is the standard response — assign the whole group to one arm, so any spillover happens inside an arm rather than across the comparison.
The point is to make the boundary of the cluster wider than the reach of the interference. A collaboration feature spills within a workspace, so workspaces are clustered. Word of mouth spreads within a friendship group, so communities are. Advertising spills across a media market, so geographic regions are — a geo experiment is cluster randomisation with markets as clusters. If the spillover crosses your cluster boundary, the design has not solved the problem, only reduced it.
The cost is severe and predictable. Users within a cluster resemble each other, so a cluster of thirty contributes much less than thirty independent observations. The design effect quantifies it — one plus the average cluster size minus one, times the intraclass correlation — and at typical values it inflates the required sample several times over. Nine hundred workspaces containing twenty-four thousand users can be worth about five thousand independent users, and no amount of extra users inside existing clusters improves that. More clusters is what buys power.
That last point governs the design. If you can choose the clustering level, prefer many small clusters over few large ones: two thousand clusters of ten beats two hundred of a hundred, even at identical total users, because the effective sample is driven by cluster count. It is also why geo experiments with a handful of markets are so underpowered — twenty markets is twenty units, whatever their population.
The analysis has to respect the clustering too, which is where implementations commonly fail. Standard errors must be computed at cluster level, either by aggregating each cluster to a single value or by using cluster-robust errors. Analysing the individuals inside clusters as independent reproduces exactly the overconfidence the design was chosen to avoid — see the analysis unit for the general form of that mistake.
One factor governs the cost and one consequence follows from it: power comes from the number of clusters, not the number of people inside them.
DEFF = 1 + ( m − 1 ) · ICCm is average cluster size, ICC the intraclass correlation. Multiply your required sample by this.
n_eff = n / DEFFAt m = 30 and ICC = 0.1, DEFF ≈ 3.9 — so 24,000 users behave like about 6,200.
as m grows, n_eff → k / ICCk is the number of clusters. Adding people to existing clusters hits a ceiling; adding clusters does not.
aggregate to cluster, or use cluster-robust standard errorsAnalysing individuals as independent undoes the design — see the sample size calculator.
A social app tests a feature that lets people react to friends' posts. Individual randomisation is impossible: a treated user's reactions are visible to untreated friends, so control sees part of the treatment. Two cluster designs are compared over the same 180,000 users.
The same users, clustered two ways, give effective samples differing by more than sevenfold — 7,230 against 54,500.
Nothing about the population changed; only the choice of cluster. Smaller clusters keep the design effect down, so 6,000 friend groups of thirty retain seven times the information of 600 communities of three hundred. The constraint is whether the smaller boundary still contains the spillover: if reactions are mostly seen by close friends, friend groups work; if the app surfaces activity across whole communities, they do not, and the larger cluster is the honest choice despite its cost. That is the real decision, and it should be settled by looking at where interaction actually happens in the product rather than by the power arithmetic. Worth noting how brutal the large-cluster option is — a design effect of 24.9 means 180,000 users buying the precision of about 7,200, which for most effects is not enough to run at all.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
Our “Geo Testing: Unlocking True Incrementality” webinar explored how teams can measure real-world impact when A/B testing isn’t possible - using geographic experiments and synthetic controls to reveal true incremental lift in marketing and product initiatives.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.