Oops I Did It Again: Common Testing Mistakes and How to Fix Them
Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.
McNemar's test calculator
For binary outcomes measured twice on the same units: users before and after, tickets scored by two systems, matched pairs. Enter the discordant counts for an exact p-value, or size the design first — which is where these studies go wrong, because the sample size is governed by the pairs that disagree, not the pairs you collect.
Calculator
| Pairs | 783 |
|---|---|
| Expected discordant pairs | 70 |
| Achieved power | 80.0% |
| Total discordance (π_d) | 0.09000 |
| Net difference (δ) | 0.03000 |
| Ratio (ψ) | 2.000 |
Collect 783 pairs and about 70 of them should disagree. Those are the only pairs the test can use: the other 713 would have looked identical under either condition and contribute nothing but bookkeeping. On that many disagreements the test detects a net shift of 0.0300 in the marginal proportions 80.0% of the time. The discordance rate is the input to protect: for a fixed net difference, systems that disagree more often in both directions need more pairs, not fewer, and an optimistic guess at it is the usual reason a paired study ends up under-powered.
Show these numbers as a table
| Power | Sample size |
|---|---|
| 3.1% | 1 |
| 20.6% | 131 |
| 36.5% | 261 |
| 50.7% | 391 |
| 62.7% | 521 |
| 72.4% | 651 |
| 79.9% | 781 |
| 85.6% | 911 |
| 89.8% | 1,041 |
| 92.9% | 1,171 |
| 95.1% | 1,301 |
| 96.7% | 1,431 |
| 97.7% | 1,561 |
Show these numbers as a table
| Net difference in the marginals | Required pairs |
|---|---|
| 0.4% | 34,882 |
| 1.2% | 4,632 |
| 2.0% | 1,731 |
| 2.8% | 897 |
| 3.6% | 547 |
| 4.4% | 368 |
| 5.2% | 264 |
| 5.9% | 198 |
| 6.7% | 154 |
| 7.5% | 123 |
| Exact p-value | 0.0043 |
|---|---|
| Discordant pairs | 78 |
| χ² p-value | 0.0046 |
| χ² | 8.0128 |
| Degrees of freedom | 1 |
| Ratio of discordant cells (b/c) | 2.000 |
| Positive rate, first measurement | 81.10% |
| Positive rate, second measurement | 77.78% |
| Difference in marginals | 4.1% |
| Pairs | 783 |
Of the 783 pairs, only 78 disagreed — 52 one way and 26 the other — and those are the whole evidence base. The pairs that agreed are invisible to this test, so the answer would be identical with ten times as many of them. That split is further from even than chance comfortably explains (exact p 0.0043), so the two measurements do not agree on how often the outcome occurs: the marginal rates differ by 0.0332. Report b and c themselves alongside that, since a reader cannot otherwise tell whether the conclusion rests on twenty disagreements or two thousand.
Show these numbers as a table
| Discordant pairs | Observed | Even split |
|---|---|---|
| First only (b) | 52 | 39 |
| Second only (c) | 26 | 39 |
When to use it
Use McNemar's test when every unit gives you two yes/no answers and you want to know whether the proportion saying yes changed: the same respondents before and after a campaign, the same tickets routed by an old rule set and a new model, a matched case-control design. The 2×2 table is not a table of two groups — it is one group cross-tabulated with itself, rows the first measurement and columns the second. Running a chi-square test of independence on it is the standard mistake, and it does not merely lose power: it answers whether the two measurements are associated, which they obviously are.
Only the discordant pairs carry information, and that is the fact worth taking from this page. A unit that said yes both times, or no both times, would have looked identical under either condition. All the evidence lives in the off-diagonal counts b and c, and the test simply asks whether b is a fair coin's share of b + c. The design consequence is hard: power depends not on how many pairs you collect but on how many discordant pairs they produce. Ten thousand units at a 2% discordance rate give two hundred informative observations, and most tools quote the ten thousand without mentioning the two hundred.
Use something else when the pairing is absent or the outcome is not binary. Two independent groups want the A/B test sample size calculator or, once the data is in, a chi-square or Fisher's exact test. Paired continuous outcomes want the paired t-test; more than two paired conditions want Cochran's Q. And if the question is whether the new option is no worse rather than different, see non-inferiority for proportions, noting that the standard version assumes independent groups.
Assumptions
McNemar's asks for remarkably little, which is one of its virtues. What it does ask for is easy to break by accident:
- The pairs are independent of one another. Inside a pair the two measurements can be as correlated as they like — that is what the design exploits. Across pairs, dependence is fatal: two tickets from the same customer are not two independent pairs.
- The pairing is structural, not retrofitted, and the outcome is the same binary definition on both occasions. Matching units after collection on variables you have inspected makes the discordant counts partly an artefact of the matching rule; a change in how "correct" or "converted" is measured between the two occasions shifts the marginals and is indistinguishable from a real effect.
- Nothing is required about how common the event is. The exact version is a binomial test on b + c trials, valid whether the outcome occurs 2% of the time or 60%. The familiar "expected counts above five" rule belongs to the chi-square approximation and has no bearing on it.
- The design still owes you a control. A significant before-and-after McNemar's says the proportion moved, not that your intervention moved it: everything else in that window sits inside the estimate. Add a concurrent control, or use a geo test or an explicit causal design — the trap our common testing mistakes post keeps returning to.
How the calculation works
The analysis tab treats the exact conditional binomial test as primary and reports the continuity-corrected chi-square alongside it, because the chi-square is what most write-ups quote and the two should be seen to agree — or to disagree, which they do when b + c is small, and then the exact one is right. Sample size uses Connor's (1987) formula, which asks for the discordant probabilities rather than the marginals. Miettinen's and Lachin's formulations condition on different things and can land roughly 10% from Connor's on identical inputs: a real disagreement, not an error, and small next to the error in your guess at the discordance rate. That argues for a pilot rather than a fourth formula, as with any [design input taken from history](/calculators/two-sample-t-test).
- The table
a = yes/yes, b = yes→no, c = no→yes, d = no/noa and d are the concordant pairs and contribute nothing. n = a + b + c + d appears nowhere in the p-value.
- Exact conditional test
b ~ Binomial(b + c, 0.5) under H₀Two-sided p = 2·P(X ≤ min(b, c)), capped at 1. Conditioning on the discordant count is what makes the null a fair coin, and why no assumption about the outcome's base rate is needed.
- Chi-square with continuity correction
χ² = (|b − c| − 1)² / (b + c), df = 1Yates's correction, reasonable once b + c is above about 25 and noticeably conservative below it.
- Effect estimates
OR = b/c, δ̂ = (b − c)/nBoth are usually wanted. The odds ratio is what the test is about, with an interval from the binomial proportion b/(b + c); the difference in marginals is what a stakeholder asks for, and needs a paired interval such as Tango's score interval.
- Sample size (Connor, 1987)
n = [ z₁₋α/₂·√π_d + z₁₋β·√(π_d − δ²) ]² / δ², δ = π_b − π_cπ_d = π_b + π_c is total discordance, δ the net difference in the marginals. For fixed δ, more disagreement in both directions means more noise and a larger n.
- Expected informative pairs
E[b + c] = n · π_dSanity-check every design against this. Below about 25, no quantity of concordant pairs will rescue the precision.
Worked example
You are replacing a rule-based support-ticket router with a model. Every ticket can be routed by both and graded correct or incorrect, so each ticket is a pair. The old router is right about 78% of the time and you expect the new one to reach 81%. A pilot says the two disagree on roughly 9% of tickets — 6 points where the model is right and the rules wrong, 3 points the other way.
About 783 tickets, of which roughly 70 should be discordant. Those 70 are the entire evidence base; the other 713 are bookkeeping.
Two comparisons make the design legible. Against an unpaired study — two independent samples, 78% against 81% — you would need something near 2,800 per arm, 5,600 in all: pairing is worth roughly seven times the data here, because grading the same ticket twice removes the variation in how hard the tickets are. It is also fragile in a way the headline hides. If the systems disagree on 16% of tickets rather than 9% while the net advantage stays at 3 points, the requirement rises to about 1,400. The pilot estimate of π_d is the input to protect.
Interpreting the output
Report b and c themselves, before any test statistic. "The model was right where the rules were wrong 52 times, and wrong where the rules were right 26 times" is a complete description of the evidence, and it keeps the sample size honest: the comparison rests on 78 tickets, whatever the backlog held. Give the odds ratio with an interval too — OR = 52/26 = 2.0, with a 95% interval of roughly 1.3 to 3.4 from the binomial proportion 52/78. It is wide, because 78 informative pairs is not many.
The exact p-value is a statement about b relative to b + c and nothing else. It does not depend on the concordant cells, so two studies with wildly different totals and identical discordant counts produce identical p-values — correct, and also why the p-value alone gives a reader no way to judge how much data stands behind it. Where the exact test and the corrected chi-square disagree, take the exact one.
Separate the two effect measures deliberately. The test is about the odds ratio b/c, but the number that reaches a slide is the difference in marginal proportions — 81% against 78%, three points. The interval on that difference must use a paired method, because the two proportions come from the same units and an unpaired standard error is the wrong quantity. Quoting a paired estimate with an unpaired interval is a common, invisible error; our note on reporting results properly is largely about pairs like this.
Finally, a non-significant McNemar's does not establish that the two conditions are the same, and with a small discordant count it establishes very little — read the odds-ratio interval and see how much it still permits. If the real question is whether the simpler option is no worse, nominate a margin in advance and frame it as a non-inferiority test rather than hoping a large p-value is read that way.
Frequently asked questions
Why does the test ignore the pairs that agree?
Because they carry no information about which condition is better. A user who converted under both variants, or under neither, would have produced the same observation whichever variant you had shown — the pair is uninformative about the comparison by construction. Conditioning on the discordant pairs is what turns the problem into a clean coin-flipping question: given that this pair disagreed, is it equally likely to have disagreed in either direction? That is the whole test.
Can't I just run a chi-square test on my 2x2 table?
Not on this table. A chi-square test of independence asks whether the row and column variables are associated. When the rows and columns are the same units measured twice, they are of course associated, and the test will confirm it while telling you nothing about whether the proportion changed. McNemar's asks the question you meant: are the two marginal proportions different? The tables look identical on the page, which is exactly why the mistake is so common.
Should I report the exact p-value or the chi-square?
The exact binomial p-value, particularly when the number of discordant pairs is below about 25, where the chi-square approximation is unreliable even with the continuity correction. Above that the two agree closely and either is defensible. It is worth knowing that the exact conditional test is somewhat conservative — its true error rate sits below the nominal level — and some authors prefer a mid-p variant that trades a little of that conservatism for power. We report the exact one because it never overstates the evidence.
I don't know my discordance rate. How do I size the study?
Run a pilot, because this is the input the answer is most sensitive to and there is no useful default. If a pilot is impossible, think about it structurally: how often could the two conditions plausibly produce different outcomes on the same unit? Then size pessimistically, since for a fixed net difference a higher total discordance always requires more pairs. Sizing on an optimistic discordance rate is the most common way a paired binary study ends up underpowered.
Why do different tools give different sample sizes for the same inputs?
Because there are several accepted formulas and they condition on different things. This calculator uses Connor's 1987 formula; the Miettinen and Lachin formulations can differ from it by around 10% on identical inputs. None of them is wrong, and the gap between them is far smaller than the uncertainty in your estimate of the discordance rate, so the practical response is to improve that estimate rather than to shop for a formula that returns a smaller number.
What if I have very few discordant pairs, or none?
With none, the test is undefined and the honest report is that the two conditions produced identical outcomes on every unit — which is informative in its own right, though it puts no useful bound on the difference. With a handful, the exact test is valid but the confidence interval on the odds ratio will be very wide, and the discreteness means certain p-values are simply unattainable. Either collect more pairs or accept that the study cannot distinguish the two conditions, and say so rather than reporting a large p-value as a finding of equivalence.
Related calculators
Further reading
Statistically speaking: How to (properly) report A/B testing results
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Four Ways to Improve Statistical Power in A/B Testing (without increasing test duration, duh)
In A/B testing, you often have to balance statistical power and how long the test takes. Learn how Allocation, Effect Size, CUPED & Binarization can help you.
Talk to the people who build these for a living
Paired designs get decisions out of a fraction of the traffic — when the pairing is real, which is the part worth checking first. A/B Testing