A Type I error is a false positive: the test reports an effect when there is none. Its long-run rate is the significance level you chose, so at the conventional five per cent, one experiment in twenty with nothing happening will still declare a winner.
αA test can be wrong in two ways, and this is the one the whole apparatus of significance testing was built to control. A Type I error is announcing an effect that is not there. Its rate is set directly by the significance level: choose alpha = 0.05 and, across experiments where the null hypothesis is true, five per cent will produce a significant result anyway. That is not a flaw in the method — it is the price, stated in advance.
The cost of a false positive is unusually easy to underestimate because it arrives late and disguised. A false negative shows up immediately as a flat result and a dropped idea. A false positive ships. It gets built on, it becomes the justification for the next three experiments, and the lift never appears in the quarterly numbers — at which point the conversation is about attribution rather than about the test that started it. Most experimentation programmes that lose credibility lose it this way.
The rate is almost always higher than the number in the spec, and never because someone edited alpha. Testing several metrics multiplies the chances; that is multiple comparisons. Checking daily and stopping on the first green reading does the same across time — a fortnight of peeks turns 5% into roughly 25%. Slicing a null result by country, device and new-versus-returning until something appears is the same arithmetic again. Each of these is a way of taking more shots at the target while charging for one.
There is a second, quieter distortion. Because a false positive has to clear the significance threshold, the ones that survive are the results noise happened to inflate — so a spurious winner does not merely exist, it typically reports an impressive effect size. Combine that with low statistical power and it gets worse: when a test can barely detect real effects, a larger share of the things that do reach significance are noise, and all of them are overstated.
The defences are unglamorous and they work. Nominate one primary metric before the test starts and treat the rest as diagnostics. Fix the sample size in advance, or use sequential testing that is designed to be monitored. Correct when you genuinely must test a family of hypotheses. And treat a surprising win the way you would treat a surprising bug report: replicate it before you build on it. A cheap A/A test or a holdback is a small price for not spending a quarter on noise.
The rate itself is a definition rather than a derivation. What is worth computing is how quickly it inflates once a test is looked at more than once.
α = P( reject H₀ | H₀ true )A long-run frequency across hypothetical repetitions, not the probability that a particular significant result is wrong.
FWER = 1 − (1 − α)^m3 metrics gives 0.14, 5 gives 0.23, 10 gives 0.40. The reason a single primary metric is worth more than any correction.
α_each = α / mCrude and always valid. Holm's step-down procedure controls the same rate with more power and is strictly better — see the chi-square test calculator for the count-data case.
FDR = E[ false positives / total positives ]Benjamini-Hochberg controls this rather than the family-wise rate. The right target when screening many metrics, where some false positives are acceptable if most findings hold.
An experiment on a new onboarding flow reports a flat primary metric: activation is 22.1% against 22.0%, p = 0.74. Before closing it out, the analyst breaks the result down by acquisition channel, device, and new versus returning — nine segments in total — and finds that paid-social users on iOS activated 4.1 points higher, p = 0.03.
p = 0.03 does not clear the corrected threshold of 0.0056. With nine segments there was a 37% chance of finding something at least this significant with nothing happening at all.
This is the single most common route to a Type I error in a product organisation, and it rarely feels like cheating — the analyst was being thorough. But the segments were chosen after seeing the data, and the effective number of chances is larger than nine once you count the segmentations that were considered and abandoned. The finding is not worthless: it is a hypothesis. The correct next step is to run a new experiment targeted at paid-social iOS users, powered for a four-point effect, and let that one decide. Building the roadmap on the segment result skips the only step that could have told you it was real.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
In this post, we will dive into the various methods for addressing multiple comparisons, examining their strengths and weaknesses, and providing guidance on when each method is most appropriate to use.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.