Plan and read the test that asks whether a new variant's conversion, retention or opt-in rate is worse by more than you are prepared to accept. Supply the margin in percentage points and the direction that counts as worse; the calculator returns the sample size, then the one-sided p-value and the confidence bound.
| Per group | 12,333 |
|---|---|
| Group 2 | 12,333 |
| Total | 24,666 |
| Achieved power | 80.0% |
| Textbook closed form, per group | 12,333 |
| Assumed rate, group 1 | 34.00% |
| Assumed rate, group 2 | 34.00% |
| Assumed true difference | 0.0% |
| Room to the margin | 0.01500 |
You need 24,666 observations in total for the study to conclude that group 1's rate is not more than 0.0150 (1.50%) *below* group 2's 80.0% of the time, if the truth is what you assumed. The quantity being powered is the 0.0150 of room between that assumption and the margin, not the margin on its own — so an assumed shortfall of even a fifth of the margin adds half again to the sample. For a change made to save money the honest assumption is usually not exact parity, and it belongs in the field above rather than in a footnote.
| Power | Sample size |
|---|---|
| 5.2% | 1 |
| 26.5% | 2,061 |
| 41.8% | 4,121 |
| 54.6% | 6,181 |
| 65.1% | 8,241 |
| 73.5% | 10,301 |
| 80.1% | 12,361 |
| 85.2% | 14,421 |
| 89.1% | 16,481 |
| 92.0% | 18,541 |
| 94.2% | 20,601 |
| 95.8% | 22,661 |
| Non-inferiority margin | Required sample per group |
|---|---|
| 0.68% | 60,008 |
| 2.70% | 3,799 |
| 4.73% | 1,243 |
| 6.75% | 610 |
| 8.77% | 361 |
| 10.79% | 239 |
| 12.82% | 169 |
| 14.84% | 127 |
| 16.86% | 98 |
| 18.88% | 78 |
| One-sided p-value | 0.0339 |
|---|---|
| Non-inferior at Δ = 0.0150 | Yes |
| One-sided 95.0% lower bound | -0.01391 |
| Two-sided 90% interval | [-0.0139, 0.0059] |
| Difference in rates | -1.2% |
| Rate, group 1 | 33.60% |
| Rate, group 2 | 34.00% |
| z | 1.8264 |
| Standard error | 0.006032 |
A higher rate is better here, so the question is whether group 1 can be ruled out as running more than 0.0150 *below* group 2. The one-sided 95.0% lower bound on the difference in rates is -0.0139, against a margin of −0.0150 — it clears, so non-inferiority is established (one-sided p 0.0339). The claim is bounded and directional, not a claim of sameness: the best estimate of the real difference is still -0.00398, and on a large base that is a real number of people even though the test passed.
| Difference in rates (p₁ − p₂) | Estimate | Interval |
|---|---|---|
| Difference in rates | -0.40% | -1.39% to 0.59% |
This is the guardrail test for rate metrics. You are shortening a signup flow, dropping a field from a form, replacing a vendor's widget with your own — the change pays for itself in speed or maintenance, and the only question is whether conversion or retention takes a hit big enough to cancel that out. Non-inferiority makes it answerable: name the hit you could absorb, then require positive evidence that the true loss is smaller.
Running a standard A/B test instead and shipping on a non-significant result is the failure this page exists to prevent. Non-significance is a statement about your sample size, not about the rates, and guardrails are structurally prone to being underpowered: usually an afterthought on a change already approved, given a slice of traffic and a fortnight. The less carefully you run one, the likelier the reassuring answer — see when being good enough is enough.
Use two-sided equivalence for two proportions when moving in the good direction would also be a red flag: an opt-in rate that jumps after a consent redesign may mean the new wording is not doing its job. Use non-inferiority for two means for continuous outcomes. If the same users saw both variants the counts are correlated, and McNemar's test is the right starting point. If the question is only how precisely you have measured one rate, use a proportion confidence interval.
Two arms of independent Bernoulli trials, a normal approximation to their difference, and three design conditions that matter more than the arithmetic:
The estimate is the plain difference in observed rates, and the null is shifted to the margin rather than sitting at zero. There is a genuine choice of standard error here that the means case does not have: the simple version substitutes the observed rates, while Farrington–Manning uses the rates that maximise the likelihood subject to the difference being exactly the margin. The second is what the sample-size formula is calibrated against. Everything below assumes higher is better.
H₀: p₁ − p₂ ≤ −Δ vs H₁: p₁ − p₂ > −Δp₁ is the new variant. The null says it is worse by at least the margin, so the change has to earn its way past — the reverse of the usual burden of proof.
z = (d + Δ) / √(p̂₁(1−p̂₁)/n₁ + p̂₂(1−p̂₂)/n₂)Unpooled: under the null the rates differ by Δ, so a pooled estimate would be estimating something the hypothesis does not claim exists.
z = (d + Δ) / √(p̃₁(1−p̃₁)/n₁ + p̃₂(1−p̃₂)/n₂)p̃₁ and p̃₂ are maximum-likelihood estimates constrained to p̃₁ − p̃₂ = −Δ, available in closed form. The default here, and the version whose type I error stays closest to nominal at moderate sample sizes.
d − z₁₋α · SE > −ΔReport this rather than the p-value: the one-sided 95% lower bound is the number a stakeholder can weigh against the saving, and it coincides with the lower limit of the two-sided 90% interval.
n = (z₁₋α + z₁₋β)² · [p₁(1−p₁) + p₂(1−p₂)] / (Δ + d₀)²d₀ is the signed difference you expect, negative if the variant will probably lose a little. Leaving it at zero produces the cheapest design and the one most likely to run out of room.
z = (Δ − d) / SE, bound: d + z₁₋α · SE < ΔFor churn, refunds, bounce and error rates: same margin, same α, hypotheses reflected. A wrong switch converts a failed guardrail into a passed one without changing anything you would notice.
Onboarding currently ends with a four-minute product video. You want to replace it with a 40-second interactive tour, cheaper to maintain and to localise. Day-7 retention is the guardrail: it sits at 34%, and product has agreed that losing up to 1.5 percentage points is an acceptable price. You are willing to assume the tour performs about as well as the video.
About 12,300 users per arm, roughly 24,700 in total. At a one-sided α of 0.025 the same design asks for about 15,700 per arm, and assuming the tour loses 0.3 points rather than nothing pushes it to about 19,200.
The third number is the one to argue about before launch. A 0.3-point assumed shortfall is a fifth of the margin and adds more than half again to the sample, because power depends on the gap between the truth and the margin, not on the margin alone. If your honest expectation is that the shorter tour will be slightly worse — and for a change made to save money it usually is — planning against exact parity is planning to run out of room. What makes guardrails expensive is that the margin people can live with is almost always far tighter than the improvement they would have chased.
The claim you have earned is bounded and directional: with 95% confidence the new rate is not lower than the old by more than Δ. That is not "the rates are the same", and it says nothing about the upside. Quote the bound and the margin together and the sentence survives being forwarded: "the 95% lower bound is −0.9 points against an agreed margin of −1.5".
Keep the point estimate beside it. If the observed difference is −0.8 points, that is your best guess at what shipping costs, and on a large base it may be substantial even though the test passed. A guardrail confirms the loss is survivable; it does not make it disappear. The gap between test-time and real-world effects is covered in why the uplift in A/B tests often differs from real-world results.
If the bound misses the margin you have not proved harm — you have failed to rule it out, and only one of those justifies abandoning the change. Usually the honest reading is that the test was too small, and the options are more traffic, a longer window, or a genuine renegotiation with the people who set the margin. What is not an option is relaxing it after seeing the bound.
Watch the drift across successive changes too. Each variant judged non-inferior to the one before gives away a further margin, and four such decisions can leave the experience meaningfully worse than the original while every test passed. Compare against a preserved baseline, not the previous winner. If you want that built into a programme rather than reinvented per test, that is what our A/B testing practice does — get in touch.
In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.
This guide aims to give you practical principles to develop A/B testing mindset. Principles that will help you survive, thrive, and systematically reach the 20-30% test success rate that creates dramatic impact in your bottom line.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
Guardrail metrics only protect you if the test behind them was sized to. We plan the margin, the power and the decision rule together. Causal Inference Analysis