Bell Statistics

What is sample ratio mismatch (SRM)?

Sample ratio mismatch is when the observed split between experiment arms differs from the intended split by more than chance allows. It means something in assignment, delivery or logging is broken, and any result from that experiment should be discarded rather than interpreted.

Also called
SRM, traffic split imbalance
Allon Korem

Written by Allon Korem

Chief Executive Officer

Last updated

In plain English

You asked for a 50/50 split and got 50.4/49.6. On two hundred users that is nothing. On two million it is a one-in-a-billion event under a fair coin, which means the coin was not fair — something between the assignment call and the analysis table treated the two arms differently. That is a sample ratio mismatch, and it is the single most decisive diagnostic in experimentation because it is a direct test of the randomization the entire causal claim rests on.

What makes it so valuable is its breadth. Almost any pipeline fault leaves a fingerprint in the arm counts: a variant that loads more slowly and loses impatient users before the exposure event fires, a redirect that drops referrer data, a bot filter whose heuristic trips more often on one experience, an app version that fails to log for a subset of users, a caching layer serving control to users assigned to treatment. None of those announce themselves anywhere else in the output, and all of them show up here.

The consequence is not that the estimate is noisier — it is that the estimate is biased in an unknown direction by an unknown amount. If the variant lost slow-connection users before they were counted, the variant's population is now systematically faster, better-connected and more engaged than control's, and it will look better whether or not the change did anything. You cannot correct for this after the fact, because you do not know which users went missing or what they would have done. The result is not weakened; it is uninterpretable.

So SRM is a gate rather than a warning. Run the chi-square test on every experiment before anyone looks at the primary metric, use a strict threshold — around p < 0.001, since you are running this on every test and do not want a stream of false alarms — and if it trips, stop. Diagnose it, fix it, and re-run. The failure mode to guard against is cultural rather than statistical: a team that finds SRM on a winning experiment and decides to look at the numbers anyway has converted a working alarm into decoration.

Diagnosis is usually a matter of narrowing. Split the counts by day, platform, browser, country and app version, and look for where the imbalance concentrates — a mismatch confined to iOS or to one release is a very different bug from one spread evenly. Compare the count at assignment against the count at exposure, since a gap between them points at delivery rather than at bucketing. What to do when you encounter SRM walks through the sequence in more detail.

The formula

A chi-square goodness-of-fit test against the intended proportions. The test is easy; the thresholds and the discipline around them are what matter.

The test
χ² = Σ (Oᵢ − Eᵢ)² / Eᵢ, df = k − 1

Oᵢ is the observed count in arm i, Eᵢ = N × intended proportion. Two arms gives one degree of freedom — run it with the chi-square test calculator.

Threshold
flag if p < 0.001

Stricter than 0.05 deliberately: this runs on every experiment, so a 5% threshold would raise a false alarm on one test in twenty and train everyone to ignore it.

Detection scales with N
χ² ≈ N · (observed ratio − expected ratio)² / (p(1−p))

At 10,000 per arm a 1% imbalance is unremarkable; at 1,000,000 it is overwhelming. Large experiments detect small faults that were always present.

Sequential monitoring
check at assignment AND at exposure

A clean split at assignment with a mismatch at exposure localises the fault to delivery or logging rather than to bucketing.

Worked example

A two-week test on a new landing page, intended 50/50. Control logged 402,150 sessions; the variant logged 397,850. The variant shows a 2.1% relative lift in conversion, p = 0.008, and the team is preparing to ship.

Intended split
50 / 50
Observed
402,150 / 397,850 (50.27% / 49.73%)
Expected per arm
400,000
χ² statistic
23.1 on 1 df
SRM p-value
0.0000015
Reported lift
+2.1%, p = 0.008

A 0.54 percentage point imbalance — about one session in 190. Under a fair split this has a probability of roughly 1.5 in a million. The experiment is invalid.

The imbalance looks trivial and is not: at 800,000 sessions, chance produces a gap this size essentially never. Something removed about 2,150 sessions from the variant, and the question that decides everything is which ones. Investigation here found the new landing page fired its exposure event after a heavier hero image finished loading, so sessions on slow connections abandoned before being counted — which means the variant's surviving population is systematically better-connected than control's, and better-connected users convert more. The 2.1% lift is consistent with that artefact alone. Note that the SRM p-value is four orders of magnitude smaller than the result's; the evidence that the experiment is broken is far stronger than the evidence that the change works.

Common misconceptions

“The split is only off by half a percent, so it does not matter.”
Significance depends on the sample size, not on how small the percentage looks. Half a percentage point on 800,000 sessions is a one-in-a-million event under a fair split. Run the chi-square test rather than eyeballing the ratio — human intuition about what counts as a large imbalance is calibrated for hundreds, not millions.
“We can just drop the extra users from the larger arm to rebalance it.”
That fixes the count and not the bias. The users missing from the smaller arm were removed by a mechanism correlated with behaviour, so the two populations already differ; removing random users from the other arm leaves that difference intact. There is no post-hoc correction, which is why SRM means discard and re-run.
“SRM only affects the split, not the metric we care about.”
The split is the symptom; the disease is that assignment or logging treated the arms differently, which means the populations are not comparable. Every metric computed on those populations inherits the bias, including the primary one, and usually in the direction that makes the variant look better.

Frequently asked questions

What p-value threshold should I use for an SRM check?
Around 0.001 is the common choice, and it is deliberately much stricter than the 0.05 used for the experiment itself. Because the check runs on every test, a 5% threshold would raise a false alarm on one experiment in twenty, and an alarm that cries wolf that often gets ignored — which is worse than not having it. At 0.001 a trip is almost always a genuine fault.
What actually causes sample ratio mismatch?
Most often the exposure event fires at a different point in the two arms, so one arm loses users who left before being counted — a slower variant, a redirect, an extra network call. Other frequent causes are bot filtering that trips more on one experience, caching or CDN layers serving the wrong variant, an app version that fails to log, and analysis filters applied to one arm but not the other. Bucketing itself is rarely the culprit.
Does SRM apply to unequal splits like 90/10?
Yes — the test compares observed counts against whatever you intended, so the expected counts are simply 0.9N and 0.1N. Unequal designs are if anything more prone to configuration errors, since the intended ratio has to be recorded correctly in both the assignment service and the analysis, and a mismatch between those two records looks exactly like a broken experiment.
How do I check SRM with more than two arms?
The same chi-square goodness-of-fit test with k − 1 degrees of freedom, comparing all arms against their intended proportions at once. If it trips, look at the per-arm contributions to the statistic to see which arm is responsible — the overall test tells you something is wrong, and the individual terms tell you where to start looking.

Related terms

  • A/B testing

    A randomised experiment on live traffic — and randomisation is the only part that makes it causal.

  • Guardrail metric

    A metric that can veto a launch but never justify one, and why that asymmetry is the whole point.

  • Outlier

    The extreme value that decides your result — and why the rule for handling it must precede the data.

  • Randomization

    The one mechanism that makes a comparison causal — and the four ways it silently fails.

  • Selection bias

    When who ends up in the data is not who you meant to study — and more data makes it worse.

Calculate it

  • Chi-square test

    Test a contingency table of counts for association — any number of rows and columns.

  • One-proportion z-test

    Test one observed rate against a fixed target — an SLA, a benchmark, a contractual floor.

  • Fisher's exact test

    The right test for a 2×2 table of small counts — exact p-values, no normal approximation.

Further reading

Knowing the term is the easy part

Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.

References