What is a randomization unit?

Before an experiment can assign anything it has to decide what it is assigning. Users are the usual answer, but sessions, devices, accounts, pages and whole geographic markets are all used, and the choice is not a detail. It determines what an independent observation is, and every standard error, confidence interval and p-value in the analysis rests on that.

User-level is the default for good reasons. It gives a consistent experience — someone does not see one variant on Monday and another on Tuesday — and it keeps each person contributing one independent observation. Session-level assignment breaks both: a returning user can be assigned differently each visit, which is confusing to experience and means their sessions are neither independent of each other nor cleanly in one arm.

The choice is forced upward when users affect each other. Colleagues in a shared workspace will notice if half of them see a different interface, so cluster randomization by account is the answer. In a marketplace where both arms compete for the same supply, no user-level split is valid at all and the unit becomes time or geography — a switchback test or a geo experiment. The rule is that the unit must be large enough that assignment does not leak across its boundary.

Going coarser costs a great deal of power and it is worth knowing the shape of the cost. Randomising 400 accounts rather than 80,000 users means the effective sample size is closer to 400, not 80,000, because users within an account behave similarly. The relevant quantity is the intraclass correlation, and even modest within-cluster similarity inflates the required sample substantially. Cluster randomisation is chosen because it is necessary, not because it is better.

The most consequential thing about the choice is the constraint it places downstream. Whatever unit is randomised is the unit the analysis must respect: randomise users and the variance calculation is over users, even when the metric counts sessions or events. Getting this wrong is the single most common source of overconfident results in web experimentation, and it is the subject of the analysis unit.

A B2B tool tests a new collaborative editing feature. Two designs are considered: randomise by user, which gives 24,000 units, or randomise by workspace, which gives 900 units averaging 27 users each. Colleagues share documents, so a user-level split would mean some people in a workspace see the feature and others do not.

The arithmetic makes the cluster design look bad and the alternative is worse, which is the usual position. If half a workspace sees collaborative editing and half does not, the ones without it still encounter shared documents created with it, so control is contaminated and the measured effect is attenuated by an unknown amount. That is a biased estimate, whereas the cluster design gives an unbiased one with wider intervals — and a wide honest interval beats a narrow wrong one. What the numbers should change is the plan rather than the design: at 6.3% detectable, this test needs either a longer run or a more sensitive metric before it is worth starting. Note also the ICC of 0.14 is the number that drives everything here, and it was measured from historical data rather than assumed. Guessing it is how a cluster-randomised test ends up half the size it needed to be.