Fisher's exact test calculator

Use Fisher's exact test on a 2×2 table whenever the counts are small enough that you would not trust a normal or chi-square approximation. The usual trigger is an expected count below about five, but the honest rule is simpler: if the answer changes depending on which approximation you picked, stop approximating. Enterprise pilots, beta cohorts, fraud investigations — anything where the interesting cell holds four rather than four thousand is this test's territory, and it is territory where most online calculators quietly return a chi-square p-value instead.

It is also perfectly valid on large tables; the reason it is not the default there is computational rather than statistical. On a conversion test of tens of thousands, the two-proportion z-test gives essentially the same answer far faster, and its Newcombe interval on the difference is more useful than an odds ratio.

Two neighbouring designs need different tests. If the same units contribute to both rows — the same users surveyed before and after, the same accounts audited twice — the table is paired and Fisher's test is wrong; use McNemar's test, which conditions on the discordant pairs. More than two rows or columns wants the chi-square test, with the small-count warning taken seriously. And for a bound on one group's rate rather than a comparison, the proportion confidence interval calculator is the right tool — at counts this small, Wilson or Clopper-Pearson rather than anything Wald-shaped.

Fisher's exact test asks less of your data than any of its neighbours, which is the point of it. What it does ask, it asks strictly.

• Every observation is independent and falls in exactly one of the four cells. Small samples make this harder, not easier: with thirty accounts it is tempting to include the same customer's two subsidiaries, and at that n a single duplicated unit moves the p-value visibly.
• Both variables are genuinely binary and were defined before the data. Dichotomising a continuous outcome at the cut point that maximises the difference is a search dressed up as a test, and far more damaging at small n than large.
• No distributional assumption at all. Nothing to check about normality, variance or expected counts — the null distribution is the hypergeometric, computed exactly. That is the whole reason to use it, and why the small-count warnings that apply to chi-square do not arise here.
• The margins are treated as fixed. This is the assumption people argue about. Conditioning on both sets of totals is what makes the calculation exact, and also why the test is conservative: discreteness leaves the achieved type-I error below the nominal 5% rather than at it. Fisher errs towards not finding things — the safer direction for a pilot, but worth knowing before you read a p of 0.06 as an absence of effect.

A B2B client ran a hands-on onboarding pilot. Sixteen new accounts got the guided programme and eight reached the activation milestone within 30 days; the twenty left on self-serve produced three. That is 50% against 15%, which looks decisive — but the whole study is thirty-six accounts.

Four defensible procedures on one table give 0.023, 0.034, 0.057 and 0.057 — two below the conventional threshold and two above. That spread is the real finding: at thirty-six accounts, "is this significant?" depends more on the convention you chose than on what the accounts did. Report the exact two-sided 0.034, because it is computed from the hypergeometric rather than approximated and minimum likelihood is what published analyses mean. But note what the interval does with the same data: an exact conditional 95% interval on this odds ratio runs from just under 1 to around 40. The pilot is compatible with the programme doing almost nothing and with it multiplying the odds of activation many times over, and no reading of the p-value narrows that.

Lead with the interval on the odds ratio, not the p-value. Small tables produce intervals so wide that the significance verdict is nearly beside the point, and quoting an odds ratio of 5.7 without the range around it is the most misleading thing you can do with this test. A ratio spanning roughly 1 to 40 supports "probably helps, magnitude unknown" and nothing stronger — a legitimate conclusion for a pilot, provided it is stated rather than dressed up.

Expect exact tests to feel stingy. The attainable p-values form a discrete set and the achieved type-I error usually sits below the nominal 5%, so Fisher rejects slightly less often than a perfectly calibrated test would — which is why a striking table can still return 0.06. That is a known property, not a flaw to be fixed by switching to whichever method gives a smaller number.

Do not read the odds ratio as a risk ratio. With outcomes as common as those above the two diverge sharply: odds five and a half times higher is an activation rate a little over three times higher, and stakeholders will hear the first number as the second. If the absolute difference is what matters — as it usually is commercially — report the two rates and the gap between them, and use the proportion confidence interval calculator for a bound on each arm.

Finally, treat a small significant table as a reason to run a real test, not a substitute for one. Effects estimated from tiny samples that cleared a threshold are systematically overstated, which is much of why the uplift in a test often differs from what shows up afterwards. Halve your expectations and size the confirmatory run against the smallest effect worth having — the trade our experimentation work spends most of its time on.