Equivalence test calculator: two means

Use an equivalence test when the conclusion you want is "these two are interchangeable" rather than "these two differ". A cheaper vendor whose handling time must not change, a refactor that should leave average basket value where it was, a shorter survey that should recover the same mean score — the decision is to swap one thing for another, and the evidence you need is that the difference is too small to care about.

That is not the question a significance test asks. Failing to reject is a statement about the power of your study, not about the world: run a two-sample t-test on twelve observations per arm and you will fail to reject almost anything. Treating p > 0.05 as proof of sameness is the most common inferential error in product analytics, and the sloppier the experiment the likelier it becomes. We work the logic through in when being good enough is enough.

Choose this page when a difference in either direction would concern you. If only one direction matters, use non-inferiority for two means — cheaper, and the honest description of most "can we ship the cheaper thing?" decisions. If the outcome is a rate, go to equivalence for two proportions. If the same units are measured twice, the pairing belongs in the analysis: start from the paired t-test and apply the margin to the mean of the differences.

TOST inherits every assumption of the t-test it is built from, and adds one of its own that is not statistical at all:

• The margin is set before you see the data. A margin chosen after looking at the observed difference is not a hypothesis test but a description of the data with a p-value attached. Write it into the plan alongside the metric and the stopping rule.
• The margin is a business decision. It is the largest difference you would accept in exchange for whatever the change buys, and that answer comes from finance or product rather than from a formula. Half a standard deviation is not a margin; "eighteen cents per session" is.
• Independence, within and between groups — unchanged from the two-sample t-test. Clustering shrinks the standard error, which in an equivalence test means falsely concluding equivalence. The error runs opposite to the direction people check for.
• Approximate normality of the two sample means, which the Central Limit Theorem covers at a few hundred observations per arm. Heavy tails remain a real problem — see what to do when data is not normally distributed. Welch's standard error is the default here, so unequal variances are already handled.

You are moving support tickets from one outsourced vendor to another. The new contract is cheaper, and the only thing that must not change is how long a ticket takes to close: within a minute of the current average is operationally invisible, beyond that customers notice. Handling time historically averages 21 minutes with a standard deviation of 6. You plan against a true gap of 0.2 minutes rather than assuming the vendors are identical.

Two things there are worth carrying away. A 0.2-minute assumed gap — a fifth of the margin — added a quarter to the sample, because equivalence power depends on the room between the true difference and the margin, not on the margin alone. And the margin has enormous leverage: tighten it to ±0.5 minutes at the same assumed gap and the requirement rises to roughly 6,850 per arm. For comparison, a conventional test powered to detect a one-minute difference would need 758. Proving sameness costs more than finding a difference of the same size.

Read the reported p-value as the weaker of two claims. Below α, both one-sided tests rejected and the data is consistent with a difference small enough to ignore — at the margin you nominated, and only at that margin. Someone who thinks half your margin is the real threshold has not been given an answer, so put the margin in the headline rather than a footnote.

Then read the 90% interval, because it says more. Entirely inside ±Δ, you have equivalence. Straddling a margin, you do not, and the interval shows which end is the problem and by how far. Wider than the margins on both sides, and the study was simply too small. A result can be both non-significant against zero and non-equivalent, which is what an underpowered study looks like from the inside.

The reverse combination catches people out too: a difference can be significant and equivalent at once. With a large enough sample, two seconds of handling time will have p < 0.001 against zero and still sit well inside a one-minute margin. That is the point — significance asks "distinguishable from zero?", equivalence asks "small enough to ignore?". Report both, as in how to properly report A/B test results.

Finally, equivalence is a claim about the average, not the distribution. Two vendors can share a mean handling time while one has a tail of three-hour tickets that generates every complaint you get. If the tail is what matters, test the tail — a quantile, or the proportion breaching a threshold via equivalence for two proportions. Pressure-testing a margin before it enters a test plan is what our causal inference work is for.