The multiple comparisons problem is that testing many hypotheses at once inflates the chance of a false positive. Each test carries its own error rate, so twenty tests at the five per cent level give roughly a sixty-four per cent chance that at least one reports a win by luck alone.
Every significance test comes with a false-positive rate you agreed to — usually five per cent. Run one test and that is your exposure. Run twenty independent tests where nothing is happening and the chance that at least one clears the threshold is 1 − 0.95²⁰, about 64%. The individual rates were never violated; there were simply more chances. This is the multiple comparisons problem, and it is arithmetic rather than a subtlety.
It arrives in a product organisation through four doors, and only the first is obvious. Testing several metrics on one experiment. Testing several variants against one control, where a four-arm test makes three comparisons. Slicing a flat result by country, device and cohort until something appears. And looking at the same test repeatedly over time, which is the same problem across days rather than across metrics — a fortnight of daily checks turns a 5% rate into roughly 25%. The last two rarely feel like multiple testing, which is exactly why they do the most damage.
There are two different things you might want to control, and confusing them leads to the wrong correction. The family-wise error rate is the probability of even one false positive across the family; controlling it is appropriate when a single wrong conclusion is costly, as it is when you will ship on the result. The false discovery rate is the expected share of your positives that are false; controlling it is appropriate when you are screening many candidates and can tolerate some noise as long as most findings hold. Family-wise control is strict and costs power; FDR control is more forgiving and scales better past a handful of tests.
Bonferroni — divide alpha by the number of tests — is the correction everyone knows, and it is always valid and never the best available. Holm's step-down procedure controls exactly the same family-wise rate with strictly more power, so there is no situation in which Bonferroni is preferable other than mental arithmetic. For screening, Benjamini-Hochberg controls the false discovery rate and stays usable at twenty or fifty metrics where family-wise control would leave you unable to detect anything. We work through which to reach for in navigating multiple comparison corrections.
The best fix is not a correction at all. Nominate one primary metric before the experiment starts, power the test for it, and decide on it alone; everything else is a diagnostic that informs the conversation and never decides it. This costs no statistical power, which every correction does, and it forces the harder question — what is this experiment actually for — to be answered before the data can influence the answer.
The first line is the whole problem. The rest are the standard responses, in increasing order of how much power they leave you.
FWER = 1 − (1 − α)^mm independent tests at level α. 3 → 0.14, 5 → 0.23, 10 → 0.40, 20 → 0.64. Correlated metrics inflate it more slowly, but never back down to α.
reject if pᵢ < α / mValid under any dependence structure, and conservative. Its only real advantage is that you can do it in your head.
sort p₍₁₎ ≤ … ≤ p₍ₘ₎; reject while p₍ᵢ₎ < α / (m − i + 1)Same family-wise guarantee as Bonferroni, uniformly more power. There is no reason to prefer Bonferroni once you have a script.
largest i with p₍ᵢ₎ ≤ (i/m)·q; reject all up to itControls the false discovery rate at q rather than the family-wise rate. The right tool for screening many metrics — see the chi-square test calculator for the count-data case.
An experiment ships a new recommendation module and the team reports on twelve metrics: purchase rate, revenue per user, add-to-cart, session length, return visits, and seven engagement counters. One comes back significant — return visits, up 3.2%, p = 0.021. Everything else is flat.
p = 0.021 survives none of the three corrections. With twelve metrics there was a 46% chance of at least one result this significant with nothing happening at all.
Note that the smallest p-value faces the same threshold under Holm as under Bonferroni — Holm's advantage appears on the second and subsequent tests, not the first — so the choice of correction does not rescue this finding. What matters more is what was not counted: the twelve metrics are the ones that made the report, and any that were computed and quietly dropped belong in the denominator too. Return visits is now a hypothesis rather than a result, and it is a reasonable one for a recommendation module. The way to bank it is a follow-up experiment with return visits nominated as the single primary metric and powered for a 3% effect.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
In this post, we will dive into the various methods for addressing multiple comparisons, examining their strengths and weaknesses, and providing guidance on when each method is most appropriate to use.
In this blog, we’ll explain why running multiple experiments simultaneously is not only feasible but also beneficial, explore its key advantages and potential challenges, and share best practices for successful implementation.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.