A Type II error is a false negative: a real effect exists and the test misses it. Its rate is beta, and one minus beta is statistical power. At the conventional eighty per cent power, a real effect of the size you designed for is missed one time in five.
βThe second way a test can be wrong: the effect is real, and the test reports nothing. The rate is beta, and it is the direct complement of statistical power — at 80% power, beta is 20%, so one in five real effects of the size you designed for goes undetected. Unlike a Type I error, nothing about the output distinguishes it from a genuinely null result. The p-value is large either way.
That invisibility is the whole problem. A false positive eventually collides with reality; a false negative simply looks like a decision. The feature is dropped, the idea is marked as tried, and the record now says it does not work. Nobody logs it as an error because nobody knows. In an organisation that runs a lot of underpowered tests, this quietly accumulates into a body of institutional knowledge that is substantially wrong, and the wrongness all points the same way — towards "nothing works here".
Beta is not a number you set directly; it falls out of four things. The true effect size, which you do not control. The sample size, which is the lever everyone reaches for. The significance level, since a stricter threshold raises beta at fixed sample. And the variance of the metric, which is the lever most teams ignore and the cheapest one available — CUPED, a less volatile primary metric, or trimming outliers all cut beta without a single extra user.
Distinguishing a false negative from a real null is not done with the p-value, which cannot tell you. It is done with the confidence interval. If a flat result comes with an interval running from −2% to +11%, an effect worth shipping is still entirely consistent with the data and the experiment has not answered the question. If it runs from −0.3% to +0.4%, you have genuinely ruled out anything material, which is a strong and useful finding. Same verdict on significance, opposite conclusion.
And if the question really is whether two things are the same — a cheaper vendor, a simplified flow, a model refresh — no amount of power will let an ordinary test deliver that answer. It needs the hypotheses inverted, which is what equivalence and non-inferiority tests do. Treating a non-significant result as proof of sameness is the single most common way a Type II error gets promoted into a stated conclusion.
Beta is defined against a specific alternative, which is why it cannot be quoted without one. The formulas below are the two-sided normal approximation used by every standard sample-size calculation.
β = P( fail to reject H₀ | H₁ true )Always conditional on a particular effect size. "The test has beta = 0.2" is meaningless without saying beta against what.
power = 1 − β80% power is beta = 0.20; 90% power is beta = 0.10. Moving between them costs about 34% more sample.
β = Φ( z₁₋α/₂ − |δ|/SE )Everything that shrinks SE — more data, less variance — shrinks beta. So does a larger true effect δ, which is why marginal changes are the ones that go undetected.
SE = √( 2σ² / n ) ⟹ halving σ² is worth doubling nThe reason variance reduction is the cheapest way to cut beta — see the A/B test sample size calculator.
A team ships a redesigned pricing page. The test runs for two weeks on 18,000 sessions per arm, against a 2.4% purchase rate. The result: 2.4% versus 2.6%, p = 0.31. The team concludes the redesign did not work and reverts it.
The test had a 31% chance of detecting the effect it was looking for, so a Type II error was the most likely outcome even if the redesign worked exactly as hoped.
The interval is the giveaway. It still contains a 0.59-point gain — a 25% relative lift, far above anything the team would have needed — so the data have not excluded success, they have failed to examine it. Reverting is the expensive mistake here, because the cost is not the two weeks: it is the entry in the record saying pricing-page redesigns do not move purchases, which will be cited the next time someone proposes one. The honest write-up is "inconclusive, powered for 31%", and the options are to run six weeks instead of two, or to apply variance reduction and run three.
In A/B testing, you often have to balance statistical power and how long the test takes. Learn how Allocation, Effect Size, CUPED & Binarization can help you.
In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.