Bell Statistics

Sample ratio mismatch (SRM) calculator

Use this calculator to check whether your observed allocation matches what you planned.

If a significant mismatch is detected, you must investigate and resolve its cause before moving forward.

Calculator

Test data

A proportion or a percentage — 0.5 or 50%.

Test parameters
Sample ratio mismatch check
Proportion of users, control (observed)—
Proportion of users, test (observed)—
Chi-square statistic—— degrees of freedom
p-value—

When to use it

Use this calculator after your A/B test has finished collecting data, but before interpreting the results. It checks whether users were actually split between groups as planned—a step that should be performed for every experiment.

A mismatch can indicate a problem with random assignment, which is the foundation for attributing differences between groups to the intervention. A significant SRM suggests that systematic differences may exist between the groups, potentially due to the randomization mechanism or its interaction with the treatment. These differences can undermine the validity of the experiment.

If you detect a significant SRM, identify and resolve its cause before proceeding with the analysis.

Assumptions

The calculation assumes:

  • Independent observations: each user is counted once, in exactly one group.
  • A sufficiently large sample: the calculation uses a chi-square approximation, which is reliable when expected counts in each group are reasonably large (typically ≥ 5). For very small samples, this check is less meaningful, since minor imbalances are common by chance.

How the calculation works

The calculator compares the observed number of users in each group to the number you would expect under your intended allocation, using a chi-square goodness-of-fit test.

Expected counts
N = Oc + Ot
Ec = N × (1 − pt)
Et = N × pt

The expectation is scaled to the total you actually observed, so this is a test of the *split* rather than of the volume. An experiment that ran at half the traffic you forecast is not an SRM, and a test built on absolute forecast counts would flag every one of them.

  • Oc and Ot are the observed user counts in the control and test groups.
  • Ec and Et are the expected counts under your specified allocation.
  • pt is the expected proportion of users in test, as entered, so the expected proportion in control is 1 − pt.
The chi-square statistic
χ² = (Oc − Ec)²Ec + (Ot − Et)²Et

Each term is a squared gap measured in units of its own expected count, which is what makes a shortfall of 300 users alarming against an expectation of 10,000 and unremarkable against one of 10,000,000. No continuity correction is applied, matching R's `chisq.test` on a goodness-of-fit call.

Degrees of freedom and the p-value
df = 1
p = 1 − Fχ²(χ², df)

One degree of freedom for a two-group comparison: the total is fixed by the data, so once you know how many landed in control you know how many landed in test. F_χ² is the cumulative distribution function of the chi-square distribution. On one degree of freedom this is exactly the two-sided one-proportion z-test — χ² is the square of that z — which is why there is no tails control on this page.

Interpreting the output

A small p-value indicates the observed group sizes are unlikely to have arisen from your intended allocation by chance alone — this points to a problem in your experiment setup (e.g., a bug in the randomization logic, users being dropped asymmetrically during data collection, or a bot/crawler disproportionately affecting one group) rather than a real treatment effect. Because SRM checks are typically run on large samples, even small allocation imbalances (e.g., 50.5% vs. 49.5% instead of an even split) can become statistically significant — which is why a stricter significance level than usual is recommended — 0.01 is the default here — to avoid over-flagging imbalances too small to matter in practice.

If an SRM is detected, we recommend investigating the cause before drawing any conclusions from the experiment's results, since a mismatch can bias observed treatment effects in either direction.

Worked example

Suppose you ran an experiment with an expected proportion of 50% of users in test, and observed 10,345 users in control and 9,655 users in test (20,000 total).

Observed users, control
10,345
Observed users, test
9,655
Expected proportion of users in test
0.5
Significance level (α)
0.01

The observed split is 51.73% control and 48.27% test. The calculator computes expected counts of 10,000 users per group. The chi-square statistic is 23.805 on 1 degree of freedom, corresponding to a p-value of 1.1e-6, which is well below α = 0.01.

This indicates a sample ratio mismatch: the observed split is significantly different from the expected 50/50 allocation. We recommend investigating your randomization, logging, or filtering logic before trusting this experiment's results.

Frequently asked questions

Why is the default significance level 0.01 rather than 0.05?
Because of how much data an SRM check usually runs on. At 20,000 users per group, a split of 50.5% against 49.5% already clears p = 0.05 — and half a percentage point is well inside what a hashing function does to a finite sample. Running the check at 0.05 on every experiment means investigating roughly one in twenty healthy tests, which is how teams learn to ignore the alarm. 0.01 keeps the false-positive rate low enough that a flag is worth acting on, while a genuinely broken split still produces p-values many orders of magnitude below it rather than just under the line.
The check flagged a mismatch. Where do I look first?
In order of how often each one is the answer: filtering, then logging, then randomization. Filtering is first because it is usually applied after assignment — a bot filter, a minimum-session-length rule, an outlier cap — and any rule that is more likely to remove a user from one group than the other produces exactly this. Logging is second: an event that fires reliably in control and intermittently in the variant loses you users you never see. Randomization itself is the rarest cause and the easiest to test, because you can run the hash over a synthetic user list offline and check the split without touching production.
My test was never meant to be 50/50. Does this still work?
Yes, and it is exactly what the expected-proportion field is for. Enter the share you configured — 0.1 for a 10% ramp, 0.33 for a third — and the control group's expectation is inferred as one minus that. The one thing to be careful about is a split that changed mid-experiment: if you ramped from 10% to 50% partway through, no single expected proportion describes the whole period, and the check has to be run separately on each phase or it will flag a ramp you did on purpose.
I ran an A/B/n test with three variants. Can I use this?
Not directly — this page is built for two groups. The chi-square calculator's goodness-of-fit mode takes any number of categories and any hypothesised split, and is the same test with more degrees of freedom; it is linked from the method section above. Set α there to 0.01 by hand, since it defaults to the usual 0.05. It is also worth running the pairwise checks afterwards if the overall test flags: with three or more groups the omnibus statistic tells you the split is wrong without telling you which group is short, and the contributions per category are what point at it.
No mismatch detected. Does that mean my experiment is valid?
It means one specific failure mode has been ruled out. An SRM check compares group sizes and nothing else, so it cannot see a bucketing bug that assigns correctly but exposes users to the wrong experience, a carryover from a previous test on the same population, or an instrumentation change that landed mid-flight. Treat a clean SRM the way you would treat a passing smoke test: necessary, cheap, and not a substitute for knowing how assignment actually works in your stack.

Related calculators

  • Chi-square test

    Test a contingency table of counts for association — any number of rows and columns.

  • A/B test analysis

    Read a finished experiment for a binary, continuous or ratio KPI — the lift, its confidence interval and a p-value.

  • A/B test sample size

    Size an experiment before you launch, for a binary, continuous or ratio KPI — each with its own design and its own assumptions.

  • MDE calculator

    Given a fixed sample size, find the smallest effect your A/B test is powered to detect — for a binary, continuous or ratio KPI.

Further reading

Talk to the people who build these for a living

We check the plumbing before the statistics, because an experiment that did not run as designed cannot be rescued by the analysis. A/B Testing

References

  • Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing Sample Ratio Mismatch in Online Controlled Experiments. KDD '19, 2156-2164.
  • Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, §3.