What is statistical significance?

Statistical significance is a decision rule, and it helps to see it as one. Before an experiment you pick a threshold — the significance level, conventionally 0.05 — which is the false-alarm rate you are willing to live with. You run the test, compute a p-value, and if it falls below the threshold you call the result significant and act as though the effect is real. That is the whole mechanism. Everything else attached to the phrase is interpretation people have layered on top.

The rule exists to stop a specific failure: eyeballing a difference and believing it. Random variation produces differences constantly, and human beings are extremely good at seeing patterns in them. A pre-registered threshold takes the judgement out of the moment where you are most motivated to reach a particular conclusion. That is a genuine achievement, and it is why the convention has survived a century of criticism it largely deserves.

What the rule cannot do is tell you whether the effect matters. Significance is a function of the effect and the sample size together, so it can be reached by a large effect on a small sample or a trivial effect on an enormous one. At ten million users per arm, a 0.02% lift — worth nothing to anybody — is significant at p < 0.001. This is why the word does so much damage in a business setting: it sounds like it means "important", and in ordinary English it does.

The failure runs in the other direction too. "Not significant" gets read as "no difference", when it usually means the experiment lacked the statistical power to resolve the question. A test that could only have detected a 20% lift, reporting p = 0.28, has told you almost nothing about whether a 6% lift exists. Reporting that as a negative result and killing the feature is a decision made on the absence of evidence rather than evidence of absence.

The practical fix is to stop treating significance as the headline. Report the confidence interval and read it against the smallest effect you would act on. The interval carries the significance verdict for free — it excludes zero exactly when p < 0.05 — and adds the one thing the verdict throws away, which is how big the effect might be. Our full argument for making that the house standard is in how to properly report A/B testing results.

Two experiments finish the same week. Test A ran on 800 users per arm and moved conversion from 12.0% to 14.4% — a 20% relative lift — with p = 0.14. Test B ran on 900,000 users per arm and moved conversion from 3.000% to 3.036%, a 1.2% relative lift, with p = 0.002.

Test B has measured a real effect with great precision, and that effect is about a thirtieth of what anyone would fund. Shipping it is fine; celebrating it is a misreading, and building a roadmap around results like it is how a team spends a year being statistically rigorous about nothing. Test A found a lift three times larger than the ship threshold and could not prove it, because 800 users per arm cannot resolve a 20% effect on a 12% baseline. Its interval still contains a 5.6 point gain. The correct action is to rerun Test A with enough traffic, and the correct summary of Test B is "real, and immaterial" — a sentence the word significant actively prevents people from writing.