What is the Bonferroni correction?

The Bonferroni correction is the simplest answer to multiple comparisons: if you are running twelve tests and want an overall 5% error rate, judge each one at 0.05 divided by 12, or 0.0042. That is the whole method. Its guarantee is that the family-wise error rate will not exceed the level you chose, and that guarantee holds regardless of how the tests relate to each other.

That last property is what keeps it in use. Most alternatives assume something about dependence between tests, or are more complicated to reason about. Bonferroni assumes nothing, works on a napkin, and cannot be got wrong. When a correction has to be explained to a room that includes people who do not want a lecture on step-down procedures, that simplicity has real value.

The cost is that it is conservative, and increasingly so as the tests become correlated. Experiment metrics usually are correlated — conversion, revenue and add-to-cart move together — so tests that are significant tend to be significant jointly, and the true family-wise rate under Bonferroni comes in well below the nominal level. You are paying for protection you already had, in the currency of statistical power.

For strict error control there is a better option that is no harder to apply. Holm's step-down procedure sorts the p-values ascending and compares the smallest against α/m, the next against α/(m−1), and so on, stopping at the first failure. It controls exactly the same quantity under exactly the same assumptions and is uniformly more powerful — never worse, sometimes considerably better. If software is doing the arithmetic there is very little reason to use plain Bonferroni instead.

And for larger, exploratory sets, the more useful question is whether family-wise control is what you want at all. Guaranteeing a 5% chance of any error across twenty diagnostic metrics leaves nothing detectable. The false discovery rate bounds the expected share of your significant results that are wrong instead, which is a weaker claim, considerably more power, and a better match for screening.

A five-arm test compares four challengers against control on the primary metric. The four p-values come back at 0.008, 0.021, 0.043 and 0.190. Three corrections are applied to the same numbers.

Holm and Bonferroni agree here because the smallest p-value fails at the same first threshold, which stops Holm's step-down immediately — Holm's advantage only materialises when the leading comparisons pass. Dunnett's is the interesting column: it finds two because it exploits something the other two ignore, namely that all four comparisons share the same control arm and are therefore correlated. Bonferroni treats them as though they could be arbitrarily related, which is safe and wastes power that the design's structure had already provided. The practical lesson is to use the correction that matches the structure of what you ran. For challengers against a common control, Dunnett's. For a general set of unrelated tests, Holm. Bonferroni's place is where the arithmetic has to be done quickly and defended simply, and it should be understood as buying that simplicity with power.