McNemar's test compares two binary measurements taken on the same subjects. It looks only at the cases that changed in one direction versus the other, discarding everyone whose answer was the same both times because they carry no information about which condition is better.
When the same subjects are measured twice on a yes-or-no outcome, the four possible outcomes are: yes both times, no both times, yes then no, and no then yes. McNemar's test looks at the last two and ignores the first two entirely. The reasoning is that a subject who converted under both conditions, or under neither, tells you nothing about which condition is better — they would have landed in the same place regardless, and including them only dilutes the comparison.
That makes it the binary counterpart of the paired t-test, and it inherits the same advantage. Between-subject variation is removed from the comparison, because each subject serves as their own control. A population where some people are eager converters and others never buy no longer swamps the signal, since the eager converters and the never-buyers both fall into the discarded agreement cells. What remains is only the people the treatment actually moved.
The discarding is what people find counterintuitive, and it is worth being precise about why it is correct rather than wasteful. Suppose 900 subjects convert under both conditions, 30 convert only under A, and 70 only under B. The question is whether B genuinely moves more people than A, and the evidence for that lives entirely in the 30 versus 70 split. The 900 are consistent with A and B being identical and equally consistent with B being far better; they simply do not discriminate. Adding them to the denominator would make the test less sensitive, not more honest.
Its natural home in experimentation is anywhere the same unit sees both conditions: interleaving experiments in search and recommendations, where one user is shown a blended result set and their click reveals a preference; before-and-after studies on a binary outcome; and diagnostic or model comparisons where two classifiers score the same set of cases. It is not applicable to a standard A/B test, where each user sees one variant and there is no second measurement to pair with.
The classical version uses a chi-square approximation with one degree of freedom, and it needs a reasonable number of discordant pairs — the usual guidance is at least 25. Below that the approximation is unreliable and the exact binomial version is the right choice, which is the same relationship Fisher's exact test has with chi-square on unpaired data. Note that the threshold applies to the discordant pairs alone, so a study with 5,000 subjects and 18 changers still needs the exact test.
Only two of the four cells appear anywhere in the arithmetic, which is the whole idea stated as a formula.
χ² = ( b − c )² / ( b + c )b and c are the two discordant counts. The concordant cells a and d appear nowhere at all.
χ² = ( |b − c| − 1 )² / ( b + c )Yates' version, still the default in some software. Conservative, and largely superseded by the exact test when b + c is small.
p = 2 · P( X ≥ max(b, c) ), X ~ Binomial( b + c, 0.5 )Under the null each discordant pair is a coin flip. Use when b + c < 25 — see the McNemar's calculator.
difference in proportions = ( b − c ) / nn is all subjects, including the concordant ones. The test ignores them; the effect size does not.
A search team runs an interleaving experiment on 4,200 sessions, showing each user results blended from two ranking models and recording which model produced the clicked result. In 3,540 sessions the click was ambiguous or both models would have surfaced the same item; 268 sessions favoured model A and 392 favoured model B.
p = 0.0000014 on one degree of freedom. Model B is preferred in 59.4% of the sessions that discriminated between the two.
The test rests on 660 sessions out of 4,200, and that is the design working rather than data being wasted. The other 3,540 sessions were genuinely uninformative about which model ranks better — both models would have produced the same click — so including them would have diluted a clear signal into a muddy one. Two cautions. The 59.4% preference is among discriminating sessions only, and translating it into a business impact needs the whole denominator: the difference across all sessions is (392 − 268) / 4,200, or 3.0 percentage points, which is the number a stakeholder should hear. And interleaving measures preference between two rankings, not whether either ranking increases revenue — a model users prefer click-for-click can still perform worse on the outcome the business cares about.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.
