The null hypothesis is the assumption a statistical test starts from: that there is no effect, no difference, nothing happening. The test measures how badly your data contradict it. You never prove it true — you either reject it or fail to.
H₀Statistical testing runs by contradiction. You assume the boring thing — the two variants convert identically, the campaign changed nothing, the coefficient is zero — and then ask how strange your data would be if that assumption held. That assumption is the null hypothesis, written H₀, and the p-value is the answer to the question. If the data are strange enough, you reject H₀ and go with the alternative, H₁, which is usually just "something is going on".
The asymmetry is deliberate and it is the part people find uncomfortable. Rejecting the null is a positive finding; failing to reject it is not the mirror image. You have not shown the null is true, only that you could not distinguish your data from it — which is exactly what an underpowered test produces whether or not an effect exists. This is why the language is always "fail to reject" rather than "accept", and the pedantry is doing real work.
How you state the null decides what the test can tell you, and the default is not always the right one. "No difference between arms" is the right null for a feature launch. It is the wrong null when the question is whether a cheaper vendor is good enough, because failing to reject it would be treated as proof of equivalence when it is nothing of the kind. That case needs the logic inverted: the null becomes "the difference is at least as large as the margin I care about", and rejecting it is what demonstrates equivalence. That is what an equivalence test does, and what non-inferiority tests do for the one-sided version.
The other thing to decide in advance is direction. A two-sided null says the effect is exactly zero and is contradicted by a difference either way; a one-sided null says the effect is at most zero and is only contradicted by a difference in one direction. One-sided is more sensitive in the chosen direction and completely blind to harm in the other, which is a trade almost nobody would accept out loud in a product setting. Choosing it after seeing the data is not a choice at all — it doubles the real error rate.
The null is also the thing being assumed when a test says nothing at all about causation. A test on observational data can reject "no association" without licensing any claim about what causes what, because the null it rejected was never about causes. Getting from association to effect requires a design where something was assigned rather than observed — randomization, a geo experiment, or an explicit causal inference model that states its assumptions.
The null is a statement about a population parameter, so it is written as one. What changes between designs is the parameter and the comparison value, and the last row is the one that surprises people.
H₀: μ₁ − μ₂ = 0 vs H₁: μ₁ − μ₂ ≠ 0The default for an A/B test on a measured outcome. Rejected by a difference in either direction — see the two-sample t-test calculator.
H₀: p₁ − p₂ = 0 vs H₁: p₁ − p₂ ≠ 0The conversion-rate version. Same logic, different standard error.
H₀: μ₁ − μ₂ ≤ 0 vs H₁: μ₁ − μ₂ > 0Blind to harm by construction. Legitimate only when the direction is fixed before the data exist and a negative effect would lead to the same action as no effect.
H₀: |μ₁ − μ₂| ≥ δ vs H₁: |μ₁ − μ₂| < δNow rejecting the null demonstrates closeness. δ is the margin you are willing to call irrelevant, and choosing it is a business decision, not a statistical one.
You are moving from an expensive payments provider to a cheaper one and need to know that checkout completion will not get worse. The old provider completes 91.0%. You decide anything within half a percentage point is acceptable, and you run the switch on 20% of traffic for three weeks: 84,900 of 93,300 sessions complete on the new provider, or 91.0%.
The conventional test fails to reject its null. The equivalence test rejects H₀: |difference| ≥ 0.5pp with p = 0.004, and the 90% interval runs from −0.24 to +0.25 percentage points.
Both tests ran on the same data and only one of them answered the question. The conventional p = 0.91 says nothing: it is what you would see if the providers were identical and also what you would see if the test were too small to tell. The equivalence test makes the claim you actually need — the difference is inside the half-point you called acceptable — and it does so by putting "they differ meaningfully" in the null and rejecting it. Which hypothesis goes in H₀ decides which conclusion is available to you, and no amount of data will let a test deliver a conclusion its null was not built to support.
In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.
Choosing between one- and two-tailed hypothesis affects every stage of A/B testing. Learn why the hypothesis direction matters and explores the pros and cons of each approach.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.