One tailed vs. two tailed tests
Choosing between one- and two-tailed hypothesis affects every stage of A/B testing. Learn why the hypothesis direction matters and explores the pros and cons of each approach.
What is the alternative hypothesis?
The alternative hypothesis is the claim an experiment is built to find evidence for, standing against the null. It is never proven — a test can only reject the null or fail to, and the alternative is what that rejection points towards.
H₁
In plain English
Significance testing works by setting up two competing statements. The null hypothesis says nothing is going on — the two variants convert identically. The alternative says something is: they do not. The test computes how surprising the data would be if the null were true, and if that is surprising enough, the null is rejected and the alternative is what remains standing.
The asymmetry between them is the part that gets lost. The test never evaluates the alternative directly; every calculation is performed under the null. So rejecting the null does not prove the alternative, it removes the competing explanation — which is a weaker and more honest claim. And failing to reject the null is not evidence for it, merely a failure to accumulate enough evidence against, which is why "no significant difference" and "no difference" are different statements.
Its form determines the shape of the test. A two-sided alternative — the rates differ, in either direction — splits the error budget across both tails and is the correct default for product experimentation. A one-sided alternative claims a direction and puts the whole budget on one side, which is more sensitive there and blind to harm in the other. That trade is argued in full under one-tailed versus two-tailed testing, and the short version is that the direction must be chosen before the data arrives or the error rate doubles silently.
It also carries a magnitude, even when nobody writes one down. Powering an experiment requires a specific alternative to power against: not merely "the rates differ" but "they differ by at least this much". That number is the minimum detectable effect, and choosing it is choosing which alternative the experiment is actually equipped to distinguish from the null. An experiment powered for a 10% lift has very little to say about a real 3% one.
The practical discipline is to state it before launching, with its direction and its magnitude, alongside the primary metric. Doing so converts a vague expectation into a testable claim and removes the possibility of adjusting it afterwards to match whatever the data produced — which is one of the more common routes into p-hacking and one of the easiest to close.
The formula
The pair of statements, in the three forms that matter, plus the version with a magnitude that a sample-size calculation actually needs.
- Two-sided
H₀: μ₁ = μ₂ H₁: μ₁ ≠ μ₂The default. Detects a difference in either direction and splits the error budget between the tails.
- One-sided
H₀: μ₁ ≤ μ₂ H₁: μ₁ > μ₂The whole budget on one side. More sensitive there and structurally unable to detect harm.
- Non-inferiority
H₀: μ_new − μ_old ≤ −Δ H₁: μ_new − μ_old > −ΔThe null is that the change IS meaningfully worse — see non-inferiority testing.
- The version with a magnitude
H₁: |μ₁ − μ₂| ≥ MDEWhat a power calculation requires. Without a magnitude there is nothing to size the sample against — see the sample size calculator.
Worked example
Two teams test the same checkout change on a 3.8% conversion rate with 90,000 users per arm. They differ only in the alternative hypothesis they registered beforehand. The observed result is a 4.6% relative lift in both cases.
Both teams see the same data and the same p-value. Only Team A can say the experiment was capable of finding this effect.
The p-value does not distinguish the two situations at all, which is why the alternative has to be registered rather than inferred afterwards. Team A designed for a 4% effect, observed 4.6%, and has a result the design was built to detect. Team B designed for 10%, which means at the effect that actually occurred their test had 22% power — so had the effect been real and this size, they would have missed it more than three times in four. Getting a significant result from an underpowered test is not reassuring; it is the situation where the winner's curse bites hardest and the estimated effect is most likely to be inflated. Team B's honest report is that they found something their test was not designed to find, and the size of it should be treated with suspicion until confirmed by a test powered for it.
Common misconceptions
- “A significant result proves the alternative hypothesis.”
- It rejects the null, which is a different and weaker claim. Every calculation in the test is performed under the null, and the alternative is never evaluated directly — it is simply what remains when the null is discarded. Proof is not available from a significance test in either direction.
- “Failing to reject the null supports the null.”
- It means the evidence was insufficient to reject it, which happens routinely when a test is underpowered regardless of whether an effect exists. Supporting the null positively requires a design built for it — an equivalence test or a non-inferiority test with a margin fixed in advance.
- “The alternative hypothesis is just 'the variants differ'.”
- That version cannot be powered against. Sizing an experiment requires a magnitude, so the operative alternative always includes one — the minimum detectable effect — whether or not anyone states it. Leaving it implicit means the experiment's actual sensitivity is decided by the traffic that happened to be available.
Frequently asked questions
What should I write down before an experiment starts?
The direction, the magnitude and the metric: two-sided, at least a 4% relative lift, on conversion rate. That is enough to size the sample and to make the analysis unambiguous afterwards. Registering it costs a couple of minutes and removes the possibility of adjusting the claim to fit whatever the data produced, which is one of the easier routes into an inflated error rate.
Should the alternative hypothesis be directional?
Two-sided for almost all product experiments. A directional alternative gives up the ability to detect harm, and a change that makes things significantly worse is information you very much want. The defensible exceptions are non-inferiority and equivalence testing, where the question genuinely has a direction and the other side is covered by the design itself.
Why is the null the one being tested rather than the alternative?
Because the null specifies an exact distribution to compute against — if the two rates are identical, the distribution of the difference follows. The alternative covers a whole range of possible effects and supplies no single distribution to work from. That asymmetry is structural to frequentist testing, and it is why Bayesian methods, which assign probabilities to both, answer a different question.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.