What is the significance level (alpha)?

Alpha is a price you agree to pay. Before the experiment you decide how often you are willing to be fooled by noise — 5% is the convention — and everything else follows from that number. The critical value the test statistic must clear, the width of the confidence interval, and the sample size you need are all derived from it. It is the one parameter in the whole procedure that is a pure judgement call, and it is the one people think about least.

The word "before" is load-bearing. Alpha only controls the false-positive rate if it is fixed independently of the data. Choosing 0.05, seeing p = 0.07 and deciding that 0.10 was defensible all along does not give you a 10% error rate — it gives you an unknown one, because the threshold has become a function of the result. The same is true of switching to a one-tailed test after seeing which way the difference went, which halves the effective threshold in the favourable direction.

The budget is also spent by things that do not look like threshold changes. Testing five metrics at 5% each gives roughly a 23% chance of at least one false positive; that is the multiple comparisons problem, and the fix is to divide the budget rather than pretend each test is the only one. Checking the result daily and stopping at the first significant reading does the same thing across time — a fortnight of daily peeks turns a 5% rate into about 25%. Slicing a flat result by country, device and cohort until something appears is the same arithmetic wearing a different hat.

Raising alpha buys statistical power at the cost of reliability, and lowering it does the reverse. That is a real trade rather than a free lunch, and it should be made against the asymmetry of the decision in front of you. A cheap, reversible change where a false positive costs an afternoon can justify 0.10. A pricing change, a migration, anything you will not revisit for two years — those deserve 0.01, and the extra sample they need is cheaper than the alternative.

One symmetry is worth holding onto: alpha and beta are the same kind of object. Alpha is the rate of claiming an effect that is not there, a type I error; beta is the rate of missing one that is, a type II error. The conventional 5% and 20% encode a specific and rarely examined belief — that a false positive is four times worse than a false negative. In a mature product where most ideas do nothing, that is probably right. In an early-stage one where the cost is a missed opportunity, it may be exactly backwards.

A team runs experiments continuously and reports every significant result to the leadership meeting. They test one primary metric per experiment at alpha = 0.05, but they also check the dashboard each morning of the fourteen-day run and stop early when the result turns green.

Nobody in this team changed alpha. They set it to 0.05, wrote it down, and never touched it — and still ran at roughly five times their stated error rate, because the budget is spent by looking, not only by deciding. Around five of their twenty quarterly wins are noise, and since early stopping selects for results that noise happened to inflate, the surviving effect sizes are overstated as well. Two fixes work: fix the sample size in advance and look once, or adopt [sequential testing](/glossary/sequential-testing) with spending boundaries, which is designed to permit continuous monitoring and charges the budget honestly as it goes.