Oops I Did It Again: Common Testing Mistakes and How to Fix Them
Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.
What is Simpson's paradox?
Simpson's paradox is when a variant wins in every segment and loses overall, or the reverse. It happens when the segments have very different base rates and the variant received a different mix of them, so the aggregate is comparing unlike populations.
In plain English
A variant converts better than control on mobile. It converts better on desktop. Pooled across both, it converts worse. Nothing is miscalculated — all three statements are arithmetically correct on the same data. This is Simpson's paradox, and it arises when the segments have very different base rates and the two arms received different proportions of them.
The mechanism is a weighted average with different weights. If mobile converts at 2% and desktop at 8%, and the variant happened to receive proportionally more mobile traffic, then the variant's aggregate is pulled towards the lower-converting segment regardless of how it performed within each. The aggregate difference is measuring the traffic mix, and the segment differences are measuring the change.
In a properly randomised experiment this should not happen, and that is the most useful thing about it. Randomisation makes the traffic mix equal across arms in expectation, so a large mix difference is evidence that something upstream is broken — a redirect failing more often on one variant, a bot filter applied after assignment, or an allocation changed mid-flight. The paradox is therefore best read as a symptom, and the first response is a sample ratio mismatch check rather than an argument about which number to report.
Where the arms are genuinely balanced, a reversal is far more often a segmentation artefact than a real finding. Slicing a result many ways will eventually produce a segment that reverses by chance, and the number of possible slices is large. A reversal that appears in one of twenty segmentations and nowhere else is what multiple comparisons predicts, not a discovery about mobile users.
When the mix difference is real and unavoidable — an observational comparison, or a quasi-experiment where you could not randomise — the segment-level numbers are the trustworthy ones and the aggregate must be standardised. Reweight both groups to a common mix, which is what post-stratification does, and report the standardised difference. The unadjusted aggregate is not a compromise between the segments; it is an artefact of composition.
The formula
The paradox is a property of weighted averages, and the condition under which it can occur is easy to state.
- The aggregate
rate_overall = Σ wᵢ · rateᵢwᵢ is the share of traffic in segment i. Different weights between arms is the entire mechanism.
- When reversal is possible
requires rate variation across segments AND w differing between armsRandomisation removes the second condition, which is why a reversal is a red flag rather than a finding.
- The standardised comparison
Σ w*ᵢ · ( rate_treatmentᵢ − rate_controlᵢ )One common weight w* applied to both arms. Removes composition from the comparison entirely.
- The first check
chi-square on the segment mix between armsIf the mix differs, fix the assignment before interpreting anything — see the chi-square calculator.
Worked example
A checkout test reports a variant losing overall. Broken out by device, the variant wins on both mobile and desktop. The team checks the traffic mix before deciding what to report.
The variant wins both segments and loses the aggregate, because it received 73% mobile traffic against control's 50%.
The mix is the finding here, not the conversion result. Control and variant were both supposed to receive the same device split and did not, by a wide margin — 88,000 mobile users against 60,000 is not something randomisation produces. That points at a real defect: most likely the variant's page failed to load on some desktop browsers, so those users dropped out before being counted. If that is what happened, the experiment is invalid rather than interesting, because the desktop users missing from the variant arm are not a random subset. The correct sequence is to run the SRM check, find and fix the cause, and rerun — not to decide whether the segments or the aggregate is the truer number. Had the mix been balanced and a reversal still appeared, the far more likely explanation would be that the segmentation was one of many tried, and the reversal is noise.
Common misconceptions
- “Simpson's paradox means you should always trust the segment-level results.”
- In a randomised experiment it usually means the randomisation failed, so neither number is trustworthy until the cause is found. The segments are the better guide only when the mix difference is genuine and unavoidable — an observational comparison — and there the right answer is a standardised aggregate rather than a list of segments.
- “It is a statistical curiosity that rarely occurs in practice.”
- Reversals appear regularly in experiment analysis, and almost always because something is broken: differential dropout, a filter applied after assignment, or an allocation changed mid-test. Treating it as a curiosity means missing the defect it is pointing at.
- “Finding a segment where the result reverses is a valuable insight.”
- Usually it is what slicing many ways predicts. Twenty segmentations give plenty of opportunity for one to reverse by chance, and a reversal appearing in exactly one is the signature of a search rather than a discovery. Treat it as a hypothesis for a test powered on that segment.
Frequently asked questions
Which number should I report when the segments and the aggregate disagree?
Neither, until you know why they disagree. In a randomised experiment the mix should be equal across arms, so a large difference means assignment is broken and the whole result is suspect — check the sample ratio first. If the mix difference is genuine and unavoidable, standardise: reweight both arms to a common mix and report that difference, which removes composition from the comparison.
Can Simpson's paradox happen in a correctly randomised test?
Only by chance, and rarely. Randomisation equalises the segment mix in expectation, so a small imbalance is possible in a small experiment and a large one is not. A reversal driven by a substantial mix difference at reasonable sample sizes is evidence of a defect rather than of randomness — differential dropout is the usual culprit.
How do I prevent it?
Check the sample ratio and the segment mix on every experiment automatically, so an imbalance is caught while the test is running rather than argued about afterwards. Stratifying on the segment at assignment time guarantees balance for small experiments where random imbalance is plausible. And pre-specify which segments you will examine, so a reversal found by slicing twenty ways is recognisable as what it is.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.