What is the analysis unit?

An experiment randomises something and then analyses something. When those are the same thing, the standard machinery works. When the analysis unit is finer — users randomised, sessions or events analysed — the observations are no longer independent, and every standard error computed from them is too small. This mismatch is probably the most common source of overconfident results in web experimentation, and it is invisible in the output.

The mechanism is straightforward once seen. A user with twenty sessions contributes twenty rows, and those twenty rows are highly correlated: a heavy user is heavy in all of them. Treating them as twenty independent observations claims twenty times the information the experiment collected from that person. Across the dataset, apparent sample size inflates by roughly the average number of rows per user, and the interval shrinks by roughly the square root of that — a factor of two or three in ordinary web analytics.

The fix is to aggregate to the randomization unit before comparing. Compute each user's total or average, then run the comparison over users. A metric like clicks per session becomes a ratio metric under this treatment, whose variance needs the delta method or a bootstrap rather than the naive formula — which is a real complication and the honest one. The alternative shortcuts, such as a mixed-effects model with a random intercept per user, are valid but heavier than most experimentation platforms want to run on every metric.

There is a subtler version worth watching for, where the analysis unit is coarser than the randomization unit. Randomise users and analyse at the account level by averaging users within an account, and you have thrown away information and created units of unequal reliability — an account with three users and one with three hundred count the same. That direction is less common and less damaging, but it wastes power rather than inventing it.

The practical diagnostic is an A/A test run many times. If the observed spread of effects across hundreds of null comparisons is larger than the standard error the platform reports, the analysis unit is too fine. That check catches the problem for every metric at once, and it is the reason A/A validation belongs in any new platform's launch checklist.

A publisher tests a recommendation module on articles read. The experiment randomises 46,000 users, who generate 318,000 sessions. The result is computed two ways: over sessions, as the analytics tool defaults to, and over users after aggregating.

The last two rows are the diagnosis rather than a coincidence — the ratio of the two standard errors matches √6.9 almost exactly, which is the signature of treating correlated rows as independent. The point estimate is unaffected, and that is what makes this error so persistent: the headline number looks the same either way, so nobody notices that only the uncertainty changed. The correct reading of this experiment is that a 2.4% lift was observed and three weeks was not enough to establish it, with an interval running from roughly −0.2% to +5.0%. The session-level version would have shipped it as a confident win. The general rule this illustrates: whenever the number of analysed rows exceeds the number of randomised units, the standard error is wrong unless something has been done about it, and the factor is predictable enough to estimate before looking.