Statistical power is the probability that a test detects an effect of a given size, when an effect of that size is really there. A test with 80% power will miss a real effect one time in five. Power is decided before the experiment runs, not after.
1 − βAn experiment can fail in two directions. It can announce an effect that is not there, and it can miss one that is. The significance level controls the first — set it to 0.05 and you accept a 5% chance of a false alarm. Power controls the second. At 80% power, an effect of the size you designed for will be detected four times out of five and missed the fifth. That missed fifth is a type II error, and unlike a false positive it produces no evidence of itself: the test simply comes back flat and everyone concludes the idea did not work.
Power is not a property of a test in the abstract. It is always power *to detect a specific effect size*, and quoting it without one is meaningless. The same experiment might have 95% power to detect a 10% lift, 45% to detect a 3% lift, and essentially none to detect a 0.5% lift. So the design question is never "is this test powered?" but "what is the smallest change worth shipping, and can this test see it?" That smallest change is the minimum detectable effect, and choosing it is a business decision that gets outsourced to statistics far more often than it should be.
Four things move power, and only one of them is usually treated as adjustable. Sample size raises it, but with square-root returns: halving the detectable effect costs four times the users. A larger true effect is easier to find, which is why big wins surface quickly and marginal ones need patience. A looser significance level raises power by lowering the bar, trading one error rate for the other. And lower variance in the outcome raises it for free — which is the entire appeal of CUPED, of trimming outliers, and of choosing a less volatile metric, all of which buy sensitivity without buying traffic.
The consequence people underestimate is what low power does to the results that *do* clear significance. In an underpowered test, the only effects large enough to cross the line are the ones noise happened to inflate — so the winners you see are systematically overstated, sometimes by a factor of two or more. This is the winner's curse, and it is the mechanism behind the familiar complaint that A/B test lifts never materialise in the P&L. We take that apart in why the uplift in A/B tests often differs from real-world results.
One rule follows from all of this: compute power before the experiment, never after it. Post-hoc power calculated from the observed effect is a deterministic function of the p-value and adds no information at all — a non-significant result mechanically yields low power, so it can never explain anything. If the test has already run and come back flat, the useful object is the confidence interval, which says directly which effect sizes remain plausible.
Power is one minus the type II error rate. For a two-sided test of two proportions or two means it has a closed form built from the standard normal quantiles, which is what every sample-size calculator inverts.
power = 1 − β = P( reject H₀ | H₁ true )β is the type II error rate. The 80% convention comes from Cohen, who reasoned that a false negative is about four times less costly than a false positive — a trade worth checking against your own situation rather than inheriting.
power = Φ( |δ|/SE − z₁₋α/₂ )Φ is the standard normal CDF, δ the true effect and SE its standard error. Everything that raises power raises the term inside Φ: bigger effect, smaller SE, looser α.
n = ( z₁₋α/₂ + z₁₋β )² · ( p₁(1−p₁) + p₂(1−p₂) ) / (p₁ − p₂)²The squared effect in the denominator is the square-root law in disguise: detecting half the effect costs four times the sample. Computed for you by the A/B test sample size calculator.
z₁₋α/₂ = 1.960 at α = 0.05; z₁₋β = 0.842 at 80%, 1.282 at 90%Moving from 80% to 90% power multiplies the required sample by about 1.34 — the cheapest reliability upgrade available on most tests.
Your checkout converts at 4.0%. Product will not ship anything below a 5% relative lift, so the effect to detect is 4.0% against 4.2%. You have 60,000 sessions per arm available in a two-week window and want to know whether that is enough, at the conventional 5% two-sided level.
With 60,000 sessions per arm the test has about 38% power to detect a 5% relative lift. Reaching 80% would take roughly 176,000 per arm — about six weeks rather than two.
This test is far more likely to miss a genuine 5% lift than to find it. Running it anyway is worse than not running it, because a flat result will be read as "the idea did not work" and the idea will be dropped. Three options are honest and one is not. Run for six weeks. Reduce variance — [CUPED](/glossary/cuped) on a pre-period conversion covariate typically cuts the required sample by 20-40% here. Or accept a larger minimum detectable effect and say out loud that the test can only find a lift of 8% or more. The dishonest option is to run the two weeks, get p = 0.31, and report that the change had no effect.
In A/B testing, you often have to balance statistical power and how long the test takes. Learn how Allocation, Effect Size, CUPED & Binarization can help you.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
CUPED is a powerful technique that not only enhances the sensitivity of your tests but also helps in resource optimization by reducing the number of subjects needed for experiments.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.