One tailed vs. two tailed tests
Choosing between one- and two-tailed hypothesis affects every stage of A/B testing. Learn why the hypothesis direction matters and explores the pros and cons of each approach.
One-tailed or two-tailed test?
A two-tailed test asks whether two groups differ in either direction. A one-tailed test asks only whether one is better, spending its entire error budget on that side and giving up the ability to detect harm in the other.
In plain English
Every significance test carries an error budget — conventionally 5%, the significance level — representing how often you are willing to claim an effect that is not there. The only question here is how to spend it. A two-tailed test splits it, 2.5% in each direction, and asks whether the two groups differ at all. A one-tailed test puts the whole 5% on one side and asks only whether the variant is better, treating any result in the other direction as unremarkable no matter how extreme.
That concentration buys real sensitivity. The critical value drops from 1.96 to 1.645, so a smaller observed difference clears the bar, and the required sample falls by roughly 20% for the same power. Those are not trivial savings, and they are the entire case for the one-tailed test. What they cost is the other tail: a variant that performs dramatically *worse* produces a non-significant result, because the test has no rejection region on that side at all. You have not merely failed to detect the harm — you have built an instrument that cannot detect it.
For product and marketing experiments that trade is almost never worth taking, and the reason is practical rather than philosophical. A change that makes conversion significantly worse is information you urgently want, both to avoid shipping it and because it usually means something is broken. A one-tailed test converts that finding into a shrug. The situations where one-tailed genuinely fits are the ones where the downside is already covered by something else — a non-inferiority framing where the question is explicitly "is the new thing not meaningfully worse", or a safety check where a separate guardrail metric is watching the other direction.
The rule that matters more than the choice itself is when the choice is made. Picking one-tailed after seeing which way the data went is not choosing a test; it is doubling your false-positive rate while reporting the original number. A result at p = 0.08 two-tailed becomes p = 0.04 one-tailed by nothing more than an author's hindsight, and nothing in the output records that this happened. It is one of the cleanest forms of p-hacking available, and it is common precisely because it feels like a technicality rather than a fabrication.
In practice: default to two-tailed, decide before the test starts, write the decision down with the sample size calculation, and treat a proposal to switch mid-flight as the red flag it is. We work through the choice at greater length in one-tailed vs two-tailed tests.
The formula
The two differ in one place only — where the rejection region sits — and every consequence in sensitivity and sample size follows from that.
- Two-tailed p-value
p = 2 · P( T ≥ |t_observed| )Counts a difference in either direction as surprising. The honest default.
- One-tailed p-value
p = P( T ≥ t_observed )Exactly half the two-tailed value when the result is in the predicted direction, and near 1 when it is not.
- Critical values at 5%
two-tailed z = 1.960 one-tailed z = 1.645A 16% lower bar to clear, which is where the sample-size saving comes from.
- Effect on required sample
n_one-tailed / n_two-tailed ≈ ( 1.645 + z_β )² / ( 1.960 + z_β )²About 21% fewer users at 80% power — see the A/B test sample size calculator.
Worked example
A team tests a new onboarding flow on activation rate and plans for a 5% relative lift at 80% power from a 12% baseline. They are considering a one-tailed test to shorten the run, on the argument that they would never ship a variant that lost. Two scenarios are then compared: the variant lifts activation by 4%, and the variant drops it by 9%.
The one-tailed test finds the modest win the two-tailed test misses, and reports the serious regression as p = 0.998 — the most reassuring number in the table, attached to the worst outcome in it.
Scenario A is the advertised benefit and it is genuine: a week saved and a real effect detected. Scenario B is the bill. A 9% drop in activation is a material regression, the two-tailed test flags it at p = 0.004, and the one-tailed test returns a p-value so close to 1 that a reader skimming the summary would conclude nothing happened. Whether the trade is acceptable turns on what else is watching. If a guardrail metric independently monitors activation for harm, the one-tailed test is defensible and the week is real. If this readout is the only thing standing between a broken flow and production, it is not — and the honest way to shorten a test is to raise the minimum detectable effect, not to remove one of its two eyes.
Common misconceptions
- “We expect the variant to win, so a one-tailed test is justified.”
- Expecting a win is not the same as being willing to ignore a loss, and it is the second that a one-tailed test actually commits you to. Almost every experiment is run because the outcome is uncertain — if the direction were known there would be little reason to test. The relevant question is not what you expect but what you would do if the variant performed significantly worse, and the answer is always that you would want to know.
- “Switching to a one-tailed test after seeing the direction is a reasonable adjustment.”
- It is a doubling of the false-positive rate, presented as a 5% test. Once the direction is known, the choice is no longer a hypothesis but a description of the data, and a p-value computed that way does not mean what it claims. This is the most common form the mistake takes, because it feels like a technical refinement rather than what it is.
- “A two-tailed test is just a one-tailed test with the p-value doubled.”
- Numerically that is true when the result falls in the predicted direction, which is why the swap looks harmless. The difference is what happens in the other direction, where a two-tailed test has a rejection region and a one-tailed test has none. The two are the same arithmetic and different instruments, and the divergence appears exactly when the answer is unwelcome.
Frequently asked questions
Is a one-tailed test ever the right choice?
Occasionally, when the question genuinely is directional and the other side is covered by something else. Non-inferiority testing is the clearest case: the question is explicitly whether a cheaper implementation is not meaningfully worse, and a one-sided bound is what that asks for. A safety check with an independent guardrail watching the opposite direction can also qualify. What does not qualify is wanting a shorter test on an ordinary A/B comparison.
How much sample does a one-tailed test actually save?
About 21% at 80% power and a 5% significance level, because the critical value falls from 1.960 to 1.645. On a four-and-a-half-week test that is roughly a week. It is a real saving, and it is worth comparing against the alternatives — accepting a slightly larger minimum detectable effect, or applying variance reduction such as CUPED — both of which shorten a test without removing its ability to detect harm.
Why is two-tailed the default in experimentation platforms?
Because it matches what an experiment is for. Most tested ideas do nothing and a meaningful minority actively hurt, so an instrument that can only register improvement is the wrong shape for the job. Defaulting to two-tailed also removes a decision that is easy to make badly under deadline pressure, and it makes results comparable across teams — a mix of one- and two-tailed readouts in one dashboard is not a like-for-like comparison.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.