What is an equivalence test?

Standard significance testing is built to detect differences and is structurally incapable of establishing sameness. Its null hypothesis is that no difference exists, and failing to reject a null never confirms it — you have not gathered enough evidence to overturn an assumption, which is a different thing from having evidence for it. This matters because "the two are basically the same" is a genuinely useful conclusion, and the ordinary machinery cannot produce it.

The reason the shortcut is so tempting is that it feels reasonable: the test came back p = 0.7, so surely the variants perform alike. But an experiment with a hundred users returns p = 0.7 whether the true effect is zero or catastrophic, so the logic rewards small samples. The less evidence you collect, the more confidently you would claim equivalence — which is a good sign the reasoning is backwards.

The fix is to invert the hypotheses. An equivalence test starts from the assumption that a meaningful difference DOES exist and requires the data to rule it out. Concretely, you nominate a band — plus or minus one percentage point of conversion, say — inside which any difference is too small to care about. Then you run two one-sided tests: one ruling out a difference below the lower bound, one ruling out a difference above the upper. Pass both and you have established equivalence. That pair of tests is the TOST procedure, and it is the standard implementation.

In practice the whole thing can be read off a confidence interval. If the entire interval on the difference sits inside your band, the two are equivalent at that level; if it pokes out either side, they are not. The relationship to a non-inferiority test is then obvious — that one checks only the lower bound, because it only cares about not being worse, while this checks both. Non-inferiority is the more common need in product work; equivalence matters when being *better* is also a problem, as with a supposedly cosmetic refactor.

The cost is sample size, and it is substantial. Establishing that a difference is smaller than some band takes considerably more data than detecting a difference of that size, because you are pinning down a location rather than merely distinguishing it from zero. Narrow bands get expensive fast. This is why equivalence testing is common in pharmaceutical bioequivalence, where regulators mandate the band, and rare in product experimentation, where teams usually settle for the cheaper one-sided question.

An engineering team refactors the checkout to remove a legacy dependency. It is meant to be behaviourally identical, and the team needs to demonstrate that rather than assume it. They agree a band of ±0.5 percentage points on a 6.2% conversion rate — anything inside that is genuinely cosmetic. The test runs on 140,000 users per arm.

This is what a clean equivalence result looks like: the interval sits comfortably inside the band with room on both sides, so the conclusion does not depend on where exactly the band was drawn. Contrast it with the ordinary reading of the same data — a two-sided test gives p = 0.52, which tells you nothing and would have been reported as "no significant difference", a phrase that means "we did not look hard enough" as often as it means what the team wanted. Two things worth noting. The interval here is a 90% one, not 95%, which surprises people: two one-sided tests at 5% correspond to a 90% two-sided interval, and using 95% makes the procedure needlessly conservative. And 140,000 users per arm is a lot for a refactor — establishing sameness within half a point genuinely costs more than detecting a half-point difference would have.