What is a non-inferiority test?

Not every change is meant to improve the metric. Migrating to a cheaper vendor, simplifying a page, retiring a legacy service, replacing a heavy model with a faster one — in each case the benefit lives somewhere other than the primary metric, and the only question is whether that metric holds up. An ordinary A/B test is the wrong instrument for this. It asks whether the variant is better, and answers "we could not tell" when the two are similar, which is precisely the answer you were hoping for and cannot act on.

The problem is that failing to find a difference is not evidence there is none. A test with 200 users will fail to detect almost anything, and reading its non-significant result as "performance held up" rewards underpowered experiments — the less data you collect, the more likely you are to conclude the new version is fine. That inversion is the specific trap non-inferiority testing exists to close.

It closes it by making you commit to a margin first. You state, before the test runs, the largest drop you would tolerate — one percentage point of conversion, say, or 2% of revenue — and the test asks whether you can rule out a loss larger than that. The evidence has to establish the absence of a meaningful decline, which is a claim that gets harder with less data rather than easier. The verdict comes from a confidence interval: if its lower bound sits above minus the margin, the new version is non-inferior.

Choosing the margin is the whole exercise and it is a business decision wearing statistical clothing. Too generous and you have licensed a real loss: a 3% margin on revenue means shipping something that might genuinely cost 3%. Too tight and the test becomes unaffordable, since the required sample grows roughly with the square of the margin — halving it quadruples the traffic you need. The honest process is to work out what the change is worth on the other side of the ledger and set the margin below it, then write the number down before any data arrives.

It is deliberately one-sided, and this is the one context where that is not a red flag. The question genuinely has a direction: you want to rule out a meaningful loss and you do not care about detecting a gain, because a gain is welcome and needs no defence. That is different from the one-tailed test applied to an ordinary comparison, where blinding yourself to harm is exactly the wrong trade. Its close relative is the equivalence test, which bounds the difference in both directions.

A team wants to replace a third-party recommendation engine with an in-house model, saving £340,000 a year. The concern is add-to-cart rate, currently 8.4%. They agree in advance that a drop of more than 0.4 percentage points would wipe out the saving, so that becomes the margin. The test runs on 95,000 users per arm.

This passes and it should not be reported as a comfortable result. The in-house model is very likely slightly worse — the point estimate is −0.13 pp and the interval sits mostly below zero — and what the test establishes is that the loss is probably not big enough to matter at the margin the team chose. Read the interval, not the verdict: a true effect at the −0.38 end would cost roughly £320,000 a year in lost carts against a £340,000 saving, which is a much less attractive trade than "non-inferiority established" implies. With this little headroom the right response is more traffic before committing, not a green tick. Note also how completely the margin drove the outcome — at 0.3 pp this test fails, at 0.5 pp it passes comfortably, and none of the data changed.