A p-value is the probability of seeing a result at least as extreme as the one you got, if the effect you are testing for does not exist. It measures how surprising your data would be under that assumption. It is not the probability that your result is wrong.
pEvery significance test starts by assuming the thing you are hoping for is not true. That assumption is the null hypothesis: the new checkout converts exactly as well as the old one, the campaign moved nothing, the two groups are the same. The test then asks a single question — if that assumption held, how often would random noise alone produce a difference as large as the one in front of me? The answer to that question is the p-value.
So a p-value of 0.03 says: if the new checkout were genuinely no better, you would still see a gap this big or bigger about three times in a hundred repeats of the experiment. That is unusual enough that most people stop believing the assumption. A p-value of 0.40 says the opposite — a gap this size turns up four times in ten by chance alone, so the data give you no reason to abandon the idea that nothing happened.
The direction of that logic is the part people lose. The p-value is computed *assuming there is no effect*. It cannot then turn around and tell you the probability there is no effect, any more than "most burglars wear gloves" tells you that most people wearing gloves are burglars. Answering the question you actually care about — how likely is it that this change works? — requires something the p-value does not contain: how plausible the effect was before you ran the test. Roughly one in three published misinterpretations of a p-value is a version of this inversion, and it is why a p just under 0.05 supports a surprising claim far more weakly than most readers assume.
Two other things are missing from it, and both matter more in practice than the interpretation debate. A p-value says nothing about how *big* the effect is — with a hundred thousand users per arm, a 0.1% lift that no one would fund clears p < 0.001 comfortably. And it says nothing about how *reliably* you could have found a real effect, which is statistical power. A large p from an underpowered test is not evidence of no effect; it is evidence that the test could never have told you either way.
This is why a confidence interval belongs next to every p-value you report. The p answers a yes-or-no question about a hypothesis nobody believed anyway; the interval answers the question the business is asking, which is how much. The two are computed from the same numbers and never disagree — a 95% interval excludes zero exactly when p < 0.05 — but only one of them tells you whether the result is worth shipping. We make the longer version of this argument in how to report A/B test results.
There is no single formula for a p-value, because it is a probability read off whichever distribution the test statistic follows — t, z, chi-square, F. The pattern is always the same: standardise the observed difference into a test statistic, then measure the tail area beyond it.
p = P( T ≥ t_observed | H₀ true )T is the test statistic under the null; t_observed is the value your data produced. Everything else is bookkeeping about which distribution T follows.
p = 2 · P( T ≥ |t_observed| )The default, and the honest one: it counts a difference in either direction as surprising. Halving it after seeing which way the result went is how a 5% error rate quietly becomes 10%.
z = (p̂₁ − p̂₂) / √( p̂(1 − p̂)(1/n₁ + 1/n₂) )The workhorse for conversion rates. The p-value is the normal tail area beyond z — see the A/B test sample size calculator.
t = (x̄₁ − x̄₂) / √( s₁²/n₁ + s₂²/n₂ )For revenue, session length and other measured outcomes. The p-value is the t tail area on the Welch-Satterthwaite degrees of freedom.
You test a new checkout flow. The control converts 2,150 of 50,000 sessions; the variant converts 2,310 of 50,000. That is 4.30% against 4.62% — a relative lift of 7.4%. You want to know whether that gap is bigger than noise would produce on its own.
p = 0.0147, two-sided. The 95% confidence interval on the absolute difference runs from 0.06 to 0.58 percentage points.
The p-value says a gap this large would turn up by chance about once in sixty-eight identical experiments where the flows were truly equivalent. That is enough to stop believing they are equivalent. What it does not say is how much better the new flow is — and the interval does: somewhere between 0.06 and 0.58 percentage points, which at the bottom end is a 1.4% relative lift and at the top end a 13.5% one. If your business case needed 5%, the p-value looks like a win and the interval says you still do not know. That gap between "significant" and "decision-ready" is the single most common way a green result leads to a bad shipping decision.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Choosing between one- and two-tailed hypothesis affects every stage of A/B testing. Learn why the hypothesis direction matters and explores the pros and cons of each approach.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.