Non-inferiority calculator: two means

Use a non-inferiority test when the change is worth making for reasons outside the metric, and the metric's job is only to confirm you are not paying too much for it. A recommendation model that costs a tenth as much to serve; a support process that frees three people. The case for shipping is already made, and the experiment exists to bound the damage: the question is not "is it better?" but "how confident am I that it is not worse by more than Δ?".

The trap is reaching for a two-sample t-test and reading a non-significant result as permission. It is not: p > 0.05 means the data did not distinguish the arms, which is also what a test that was too small or too noisy produces. The incentives make it worse than the average inference error, because the team wants to ship and an inconclusive test looks like a green light. Reframing so that shipping requires positive evidence is the point, argued in when being good enough is enough.

Choose non-inferiority over an equivalence test when a surprise in the good direction genuinely needs no protection. That is the common case and it buys a cheaper design, since the whole error budget goes on one side. Go two-sided when being unexpectedly better is itself a warning: a support bot that closes tickets far faster than a human may not be solving them. For a rate, use non-inferiority for two proportions. The framing is the one in one-tailed vs two-tailed tests, with the null shifted to the margin rather than sitting at zero.

The distributional assumptions are the t-test's. The ones that actually sink non-inferiority studies are about the design:

• You have set the direction correctly. The explicit higher is better / lower is better switch exists because the two hypotheses are mirror images and the wrong one yields a confident, precisely inverted conclusion — a small p-value pointing the wrong way. Revenue and retention are higher-is-better; latency, churn and handling time are lower-is-better.
• The margin was fixed before the data existed, and it is a commercial quantity: the loss you would knowingly accept in exchange for the saving, in the units of the metric. If nobody will commit to a number in advance, that is worth discovering before you run anything.
• Independence and approximately normal sample means, as in the two-sample t-test. Long-tailed money metrics are the usual difficulty — see what to do when data is not normally distributed — and a handful of large orders can carry a claim on their own.
• The comparison is a fair one. Non-inferiority is uniquely vulnerable to sloppiness: contamination, broken assignment or a noisy outcome all pull the observed difference towards zero and therefore towards your desired conclusion. In a superiority test sloppiness costs power; here it buys the answer you wanted.

You want to replace a heavyweight recommendation model with one that costs a tenth as much to serve. Revenue per session is the guardrail, higher is better, and finance agreed in advance that a drop of up to 25 cents is worth the saving. The test ran three weeks at a 50/50 split, 60,000 sessions per arm.

The headline is not that the cheap model performed the same — it lost eight cents a session, and that point estimate is the best guess at what shipping will cost. The claim is narrower: the data rules out a loss as large as 25 cents with 95% confidence, so the saving survives the worst case the evidence permits. The conventional comparison gives a two-sided p of 0.32, which would have been reported as "no difference" and justified nothing. Now flip the direction switch by mistake: the statistic becomes (0.25 + 0.08) / 0.080 = 4.11 and p collapses below 0.0001. Here that happens to land on the same decision, but with a 40-cent drop the correct test returns p = 0.97 — a clear failure — while the flipped one still returns p below 0.0001.

Report the confidence bound, not the p-value. "The 95% lower bound on revenue per session is −$0.21, against an agreed margin of −$0.25" is a sentence a finance director can act on. "p = 0.017" is not, and it invites the reading that the model was shown to be as good — a stronger claim than the data supports.

Keep the point estimate in view for that reason. Non-inferiority answers a question about the worst case and says nothing about the expected one, which here is a real if tolerable loss. Multiply it by annual sessions before you celebrate. Our note on why the uplift in A/B tests often differs from real-world results applies with equal force to declines.

A failed test is not proof of inferiority. If the bound lands at −$0.31 against a margin of −$0.25, you have not shown the new model is worse by more than the margin; you have failed to show it is not, which usually means the study was too small. The remedy is more data or a properly renegotiated margin — not a rerun with the direction switch flipped.

Finally, be careful about stacking claims. Each variant judged non-inferior to the last can drift a full margin from the original, and after four such decisions the live system may be materially worse than the one you started with while every test passed. Clinical statisticians call it biocreep; anchor to a preserved baseline. That discipline is what our causal inference work and most of our case studies are about.