What is family-wise error rate?

A single test at the 5% level has a 5% chance of a false positive. Run twelve independent tests at that level and the chance that at least one comes back significant when nothing is happening is not 5% but 46%. The family-wise error rate is that overall figure — the probability of at least one false alarm across a set of tests — and the gap between it and the per-test rate is what makes multiple comparisons a problem rather than a technicality.

Controlling it means adjusting the per-test threshold so the overall rate lands where you intended. The Bonferroni correction is the simplest route: divide the target rate by the number of tests. Holm's step-down procedure controls the same quantity with uniformly more power and is essentially free to implement, so there is little reason to prefer plain Bonferroni once you know it exists.

The question that decides everything here is what counts as a family, and it is a judgement rather than a calculation. Every test the organisation runs this year is not a family in any useful sense — correcting at that scale would make nothing significant. The workable convention is the set of tests that bear on one decision: the metrics in one experiment's readout, or the arms of one multi-arm test. Draw it too wide and you lose all power; too narrow and the correction stops meaning anything.

FWER control is strict by design, and that strictness is appropriate when any single false positive is costly — a safety claim, a regulatory submission, or a change that would be expensive to reverse. It becomes counterproductive on an exploratory panel of twenty diagnostic metrics, where insisting on a 5% chance of any error at all leaves you unable to detect anything real.

That is where the false discovery rate comes in as the alternative. Instead of bounding the probability of any error, it bounds the expected proportion of your significant findings that are wrong — a weaker guarantee with substantially more power, and the better fit for screening. The choice between them is about consequences: FWER when one mistake matters, FDR when you are prioritising a list.

An experiment reports one primary metric and eleven secondary metrics. The primary is flat. Two secondaries come back with p-values of 0.011 and 0.038. The team wants to know whether either is a finding, and applies three approaches to the same twelve p-values.

The 46% row is the one to sit with: before any data was collected, this readout had nearly a coin-flip chance of producing at least one significant secondary on a change that did nothing. Observing two is unremarkable. All three corrections agree here, which is worth noting because they often do not — Holm and Benjamini-Hochberg are usually more permissive than Bonferroni, and the fact that even the most powerful of them rejects both p-values says the evidence is genuinely thin. What should happen next is that the 0.011 result, if it is substantively interesting, becomes the pre-registered primary metric of a follow-up experiment. What should not happen is reporting it with its uncorrected p-value and a note that it was exploratory, which is how a 46% chance of noise becomes a line in a quarterly review.