“No plan survives first contact with the enemy.”
Fortunately, A/B testing is not combat, and allocation plans tend to hold up just fine. Still, when a sample ratio mismatch (SRM) occurs between the planned and observed allocation, it can be a sign that something went wrong with the experiment setup or randomization.
Why does this matter? Random allocation is what allows us to attribute differences between groups to the treatment, rather than to other systematic differences between them. A significant SRM can undermine this assumption, making it difficult to determine whether the observed pattern is due to the treatment or to differences between the groups.
That's why checking for SRM should be the first step in your analysis. If a significant mismatch is detected, you must investigate and resolve its cause before moving forward. Use this calculator to check whether your observed allocation matches what you planned. It's a quick way to catch potential issues before you move on to the rest of your analysis.
Enter your numbers on the left and the result appears here.
Use this calculator after your A/B test has launched and started collecting data, and before you interpret its results. It checks whether users were actually split between groups the way you intended — a step worth doing on every experiment, since a mismatch here can invalidate or bias your downstream analysis, regardless of how significant your effect looks.
It is deliberately the cheapest check you can run. Two numbers off the experiment dashboard and the traffic allocation you configured, and you know within seconds whether the rest of the analysis is worth doing. Nothing downstream repairs a broken split: a difference in outcomes between two groups that were not formed at random is not evidence about a treatment, and no amount of statistical machinery makes it so.
The same arithmetic in its general form lives on the chi-square calculator — an SRM check is a goodness-of-fit test, and that page handles any number of categories against any hypothesised split. This one exists because the two-group case is what experimenters actually have, and because it needs a much stricter default threshold than a general goodness-of-fit test does.
If the split is clean, go on to the A/B test analysis calculator. If you have not launched yet, the sample size calculator is the page you want instead.
The calculation assumes:
The calculator compares the observed number of users in each group to the number you would expect under your intended allocation, using a chi-square goodness-of-fit test. Let O_c and O_t denote the observed user counts in the control and test groups, and E_c and E_t the expected counts under your specified allocation. Let p_t denote the expected proportion of users in test, as entered, so the expected proportion in control is 1 − p_t.
N = O_c + O_t, E_c = N × (1 − p_t), E_t = N × p_tThe expectation is scaled to the total you actually observed, so this is a test of the *split* rather than of the volume. An experiment that ran at half the traffic you forecast is not an SRM, and a test built on absolute forecast counts would flag every one of them.
χ² = (O_c − E_c)² / E_c + (O_t − E_t)² / E_tEach term is a squared gap measured in units of its own expected count, which is what makes a shortfall of 300 users alarming against an expectation of 10,000 and unremarkable against one of 10,000,000. No continuity correction is applied, matching R's `chisq.test` on a goodness-of-fit call.
df = 1, p = 1 − F_χ²(χ², df)One degree of freedom for a two-group comparison: the total is fixed by the data, so once you know how many landed in control you know how many landed in test. F_χ² is the cumulative distribution function of the chi-square distribution. On one degree of freedom this is exactly the two-sided one-proportion z-test — χ² is the square of that z — which is why there is no tails control on this page.
Suppose you ran an experiment with an expected proportion of 50% of users in test, and observed 10,345 users in control and 9,655 users in test (20,000 total).
The observed split is 51.73% control and 48.27% test. The calculator computes expected counts of 10,000 users per group. The chi-square statistic is 23.805 on 1 degree of freedom, corresponding to a p-value of 1.1e-6, which is well below α = 0.001. That p-value is shown in scientific notation rather than floored at four decimal places as it is elsewhere on the site — at this threshold its order of magnitude is the thing worth reading.
This indicates a sample ratio mismatch: the observed split is significantly different from the expected 50/50 allocation. We recommend investigating your randomization, logging, or filtering logic before trusting this experiment's results. Note how small the imbalance is in absolute terms — 345 users either way, 1.7% of the traffic — and how decisively it is rejected. That is what 20,000 users buys, and it is the reason this check runs at α = 0.001 rather than 0.05.
A small p-value indicates the observed group sizes are unlikely to have arisen from your intended allocation by chance alone — this points to a problem in your experiment setup rather than a real treatment effect. The usual culprits are a bug in the randomization logic, users being dropped asymmetrically during data collection, and a bot or crawler disproportionately affecting one group.
Because SRM checks are typically run on large samples, even small allocation imbalances — 50.5% against 49.5% instead of an even split — can become statistically significant. That is why a stricter significance level than usual is recommended, 0.001 being a common choice, to avoid over-flagging imbalances too small to matter in practice.
If an SRM is detected, investigate the cause before drawing any conclusions from the experiment's results, since a mismatch can bias observed treatment effects in either direction. The direction is not predictable from the imbalance itself: whatever dropped or duplicated users is very unlikely to have done so evenly with respect to the metric, which is precisely why the comparison stops being a comparison.
A clean result is not a guarantee that assignment was sound — it is a guarantee that assignment was not broken in the one way this check can see. Bucketing that is correct at the allocation point but leaks at the exposure point passes an SRM check and still invalidates the experiment, which is a separate thing to look for.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.
We check the plumbing before the statistics, because an experiment that did not run as designed cannot be rescued by the analysis. A/B Testing