What is bucketing?

Bucketing is the mechanism that puts each unit into a variant. It sounds like plumbing and it is the foundation everything else rests on: if assignment is not what it claims to be, every downstream statistic describes something other than the experiment you thought you ran. Three properties are required, and each fails in a characteristic way.

It must be random with respect to everything that predicts the outcome. That is what makes the arms comparable and is the entire basis for reading a difference as an effect. Assignment by user id parity, sign-up order, or anything else with structure will correlate with tenure or behaviour and produce a confounded comparison that looks like a clean one.

It must be deterministic for a given unit. The same user must land in the same variant on every request, every session and every device, or their experience flickers and their data belongs to both arms. This is why hash-based assignment is the standard implementation — it computes the bucket from a stable identifier rather than storing a decision, so it is consistent without needing a lookup and works identically across services.

And it must be independent across concurrent experiments. This is the property most often broken, by hashing only the user id rather than the user id combined with the experiment id. When that happens, the same users land in the same relative position in every experiment: a user in the top decile of the hash space is in variant B of everything. Assignments become correlated across tests, so interaction effects stop being random noise and become systematic, and two experiments running together can be measuring each other.

The verification is straightforward and belongs in the platform rather than in anyone's memory. A sample ratio mismatch check on every experiment catches uneven splits and mid-flight allocation changes; a periodic A/A test catches the subtler failures in metric computation and variance. Both are cheap, both run automatically, and between them they cover most of what can go wrong here.

A platform hashes only the user id, not the user id plus experiment id. Three experiments run concurrently over the same 200,000 users. Each reports a clean sample ratio and each looks healthy in isolation. An analyst checks how assignments relate across the three.

Each experiment passes its own health check because each split is genuinely 50/50 — the failure is invisible from inside any single test. What has actually happened is that the three experiments are running on two groups of users rather than eight, so no combination of treatments is ever observed except all-control and all-treatment. If any two of the three interact, that interaction is baked into all three results and attributed to whichever change the reader is looking at. The fix is one token in the hash input. The reason this is worth a worked example is that it produces no error, no imbalance and no alert: the only way to find it is to check the cross-tabulation between concurrent experiments, which almost nobody does unless they know to look. Adding a periodic assignment-independence check alongside the SRM check is cheap insurance against a class of failure that otherwise persists indefinitely.