Randomization is assigning each unit to a treatment group by chance rather than by choice. It makes the groups comparable on every characteristic at once, including ones nobody measured, which is what licenses a causal reading of the difference between them.
Randomization is the whole reason an experiment can say anything about cause. When you assign users by coin flip, the two groups end up statistically identical in every respect — device, tenure, country, intent, mood, and the fifty things you have never measured or thought of. Matching on observed characteristics can balance the ones you listed; only randomisation balances the ones you did not. So when outcomes differ by more than chance allows, the change is the only remaining explanation.
That is a stronger property than it sounds. Every non-experimental method — matching, regression adjustment, propensity score matching — controls for confounding variables you can name and measure, and is defenceless against the ones you cannot. Randomisation handles the unknown unknowns by construction, which is why it remains the only design that needs no argument about which variables were included.
In practice it fails in four ways, none of which announce themselves. The assignment can be applied at the wrong moment, so users are bucketed before they reach the surface being changed and the effect is diluted across people who never saw it — we take that apart in when allocation point and exposure point differ. It can be applied at the wrong unit, so one person straddles both arms across devices or sessions. The hashing can be biased, correlated with a previous test, or reset by a cookie policy. And the arms can be filtered differently after assignment — excluding bot traffic from one arm using a rule that fires more often in the other quietly re-introduces selection.
The routine check is a sample ratio mismatch test, and it should run on every experiment before anyone reads the result. If you asked for 50/50 and got 50.4/49.6 on two million users, that gap is far outside chance and the randomisation is broken somewhere. This check has an unusual property: it is sensitive to almost any pipeline fault at all — assignment, logging, filtering, redirects — which makes it the cheapest single diagnostic in experimentation.
Two refinements are worth knowing. Stratified assignment randomises within blocks — by country, platform, or a pre-period value of the metric — which guarantees balance on those specific dimensions instead of merely making it likely, and reduces variance at the same time. And when the unit that must be randomised is not the user, the design changes shape entirely: clusters for a marketplace where users interact, regions for a geo experiment where the treatment cannot be hidden, or time periods in a switchback where the same units alternate.
The mechanics are a hash, and the checks are two standard tests. What matters is what the hash is computed over and when.
bucket = hash(unit_id + experiment_salt) mod 100The experiment-specific salt is essential: without it, the same user lands in the same bucket in every test and effects from one contaminate the next.
E[Y(1) | assigned to treatment] = E[Y(1)]Potential outcomes are independent of assignment, so the difference in group means is unbiased for the average treatment effect. This equality is the entire causal claim.
χ² = Σ (observed − expected)² / expected, df = k − 1Flag below about p = 0.001. Use the chi-square test calculator and run it before reading any result.
t = (x̄₁ − x̄₂) / √( s₁²/n₁ + s₂²/n₂ )On a pre-experiment covariate this should be non-significant. Testing many covariates re-introduces multiple comparisons, so treat it as a smell test rather than a gate.
A test on a new checkout step is assigned when a user lands on the site, but only 18% of sessions ever reach checkout. The experiment runs to 400,000 sessions per arm. The true effect on users who see the new step is a 4% relative lift in completion.
Analysed on everyone assigned, the test has roughly a 12% chance of detecting the effect. Analysed on the 18% who actually reached checkout, it has 74%.
The randomisation is perfectly sound in both analyses — what changed is which population it is being read over. Including 82% of users who could not possibly have been affected does not bias the estimate, it dilutes it, and dilution costs power exactly as if you had shrunk the effect. The fix is to trigger the analysis at the point of exposure: count only users who reached the checkout step, in both arms. The critical detail is that the trigger condition must be evaluated identically in control — you need to know which control users *would* have hit the step — which is why exposure logging has to be instrumented deliberately rather than inferred afterwards. Filtering the variant on "saw the new step" and the control on nothing is not a fix; it is how a valid randomisation gets thrown away.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.