What is Fisher's exact test?

A chi-square test does not compute the probability of your table. It computes a statistic and compares it against a continuous curve that the statistic only approximately follows — an approximation that is excellent when the counts are large and poor when they are not. Fisher's exact test skips the approximation. It asks directly: holding the row and column totals fixed, how many possible tables are at least as extreme as the one I observed, and what fraction of the total do they represent?

The mechanism is combinatorial rather than distributional. With the margins fixed, a 2 × 2 table has exactly one free cell, so every possible outcome can be listed and its probability computed from the hypergeometric distribution. Sum the probabilities of the tables as extreme or more extreme than yours and that sum is the p-value. Nothing is estimated and nothing is approximated, which is where the word "exact" comes from — it describes the computation rather than any claim to superior accuracy about the world.

That is why it is the standard answer when a cell expects fewer than five. Below that threshold the chi-square approximation misbehaves in the dangerous direction — it tends to return p-values that are too small, so a false positive is more likely than the number claims. A test with 90,000 users and eleven conversions between the arms is precisely this situation, and it happens far more often than the raw sample size suggests, because rare events stay rare no matter how much traffic you pour in.

It has two real costs. The first is computational: enumeration grows quickly with table size, which is why it is routine for 2 × 2 and unusual for anything much larger. The second is that it is conservative. Conditioning on the margins — treating both sets of totals as fixed when in reality only one was — means the achievable p-values come in discrete jumps, and the test's actual false-positive rate sits below the nominal 5% rather than at it. You are trading some power for a guarantee that the rate is never exceeded.

In an experimentation context the practical rule is simple. Check the smallest expected count before choosing: at five or above use chi-square or a z-test and enjoy the extra power; below it use Fisher's and accept the conservatism. What you must not do is run both and report whichever crossed the threshold, which converts a principled choice into a multiple comparisons problem with a sample size of one.

An enterprise onboarding test runs on a small population: 48 accounts in control and 46 in the variant. Control produced 2 activations, the variant 8. That is 4.2% against 17.4%, a fourfold difference, and the team wants to know whether 94 accounts can support the claim.

The spread across those three numbers is the real lesson. This table sits right on the boundary where the approximation starts to matter: the smallest expected count is 5.1, barely clearing the rule of thumb, and the three methods disagree by more than the width of the decision. Anyone free to pick among them can produce whichever verdict they prefer, which is why the choice has to be made before the data is seen. Fisher's is the defensible default here, and its p = 0.049 should be read as what it is — a result that has cleared a line by a hair on ten activations in total. The honest summary is that the variant looks considerably better and 94 accounts cannot establish by how much: the confidence interval on the difference runs from roughly 1 to 27 percentage points.