Two conversion rates and a claim that they are close enough to treat as the same. This calculator sizes that study and then runs it: two one-sided tests against a margin in percentage points, the p-value that is the larger of the two, and the 90% interval that has to fall inside the margins for the claim to hold.
| Per group | 19,118 |
|---|---|
| Group 2 | 19,118 |
| Total | 38,236 |
| Achieved power | 80.0% |
| Assumed rate, group 1 | 12.80% |
| Assumed rate, group 2 | 12.80% |
| Assumed true difference | 0.0% |
You need 38,236 observations in total — 19,118 in each arm at this split — for both one-sided tests to reject 80.0% of the time, and so to conclude the two rates differ by less than 0.0100 (1.00% in absolute terms). That is far more than an ordinary A/B test on the same base rate would ask for, because the margin is usually much smaller than the lift such a test is powered to find, and the requirement grows with the inverse square of it: halve the margin and the traffic roughly quadruples.
| Power | Sample size |
|---|---|
| 0.0% | 1 |
| 0.0% | 3,191 |
| 3.7% | 6,381 |
| 33.0% | 9,571 |
| 54.4% | 12,761 |
| 69.6% | 15,951 |
| 80.1% | 19,141 |
| 87.1% | 22,331 |
| 91.7% | 25,521 |
| 94.8% | 28,711 |
| 96.7% | 31,901 |
| 98.0% | 35,091 |
| Equivalence margin | Required sample per group |
|---|---|
| 0.64% | 46,673 |
| 1.36% | 10,305 |
| 2.08% | 4,402 |
| 2.81% | 2,428 |
| 3.53% | 1,536 |
| 4.25% | 1,059 |
| 4.97% | 774 |
| 5.69% | 590 |
| 6.42% | 465 |
| 7.14% | 376 |
| TOST p-value | 0.1907 |
|---|---|
| Equivalent at ±0.0100 | No |
| 90% interval (1 − 2α) | [-0.0157, 0.0087] |
| Difference in rates | -2.7% |
| Rate, group 1 | 12.45% |
| Rate, group 2 | 12.80% |
| p, against the lower margin | 0.1907 |
| p, against the upper margin | 0.0345 |
| z, lower | 0.8752 |
| z, upper | -1.8178 |
| Standard error | 0.007427 |
The 90% interval on the difference in rates runs from -0.0157 to 0.00872, which reaches below −0.0100 of ±0.0100 (1.00%), so equivalence is not established (TOST p 0.1907). This is not evidence that the two rates differ — an ordinary comparison on these same counts may well be nowhere near significant. It means a change you said you would care about has not been ruled out, and the usual remedy is more data rather than a softer conclusion.
| Difference in rates (p₁ − p₂) | Estimate | Interval |
|---|---|---|
| Difference in rates | -0.35% | -1.57% to 0.87% |
Reach for this when a rate has to stay put. You are migrating to a new payment provider and checkout conversion must not move; you are replacing a bloated consent banner and the opt-in rate must hold; you are cutting a step out of signup and day-7 activation must survive it. Shipping is the default, and the experiment exists to rule out damage in either direction larger than a number of percentage points you can name.
A standard A/B test cannot deliver that. Its machinery is built to detect a difference, and its failure mode — p above 0.05 — is silent about whether the rates are close or the test was too small. Migration tests are where this bites: they tend to be run late, in a hurry, on a fraction of traffic, and then read as a clean bill of health. We make the same argument from the other side in common testing mistakes and how to fix them.
Use this page when both directions matter. When only the downside does, use non-inferiority for two proportions, which spends its whole error budget on the side you care about. For a continuous outcome such as revenue per session, use equivalence for two means. If the same users saw both variants, the counts are paired and belong in McNemar's test. And if you only need to know how precisely one rate has been measured, a proportion confidence interval is simpler.
The arithmetic is a normal approximation to a difference of two binomials, so the usual proportion caveats apply — plus two that are specific to margins:
Both tabs work on the absolute difference in rates, d = p̂₁ − p̂₂, and its unpooled standard error. Unpooled is deliberate: the pooled estimator used by a standard two-proportion test assumes the rates are equal under the null, and here the null is that they differ by exactly the margin. Equivalence is concluded when both one-sided tests reject at α, and we report the larger p-value alongside the (1 − 2α) interval — 90% when α = 0.05 — the interval that agrees with the procedure.
d = p̂₁ − p̂₂, SE = √(p̂₁(1−p̂₁)/n₁ + p̂₂(1−p̂₂)/n₂)Each arm contributes its own variance. No pooling, because the hypothesis under test is a non-zero difference.
z₁ = (d + Δ) / SE, z₂ = (Δ − d) / SEz₁ asks whether the difference clears the lower margin, z₂ whether it stays below the upper one. Both are upper-tail tests at the full α.
p = max(p₁, p₂), equivalent when p < αTaking the maximum keeps the overall type I error at α without correction: to declare equivalence you must clear both hurdles, so the harder one governs.
d ± z₁₋α · SE (90% at α = 0.05)Equivalence holds exactly when this lies inside (−Δ, +Δ). The 95% interval you would report beside an ordinary A/B result is the wrong one here and will occasionally appear to contradict the p-value.
n = (z₁₋α + z₁₋β/₂)² · [p₁(1−p₁) + p₂(1−p₂)] / Δ²The half-beta again: power must be delivered against both tests. Substituting z₁₋β — as though only one had to reject — understates the sample by about a fifth at 90% power and more than a quarter at 80%.
n = (z₁₋α + z₁₋β)² · [p₁(1−p₁) + p₂(1−p₂)] / (Δ − |d₀|)²Set d₀ to the shortfall you genuinely expect. The distance enters squared, so even a small expected gap moves the answer sharply, and power tends to zero as d₀ approaches the margin.
You have moved checkout to a new payment provider behind a 50/50 split and want to confirm conversion has not shifted by more than a percentage point either way — the threshold at which the integration saving stops covering lost revenue. The margin went into the plan a fortnight before launch. After a week you have 4,000 sessions per arm: 512 conversions on the incumbent, 498 on the new provider.
Difference −0.35pp, standard error 0.74pp. The lower one-sided test gives p = 0.19 and the upper gives p = 0.03, so the reported TOST p-value is 0.19 and equivalence is not established. The 90% interval is [−1.57pp, +0.87pp], which crosses the lower margin.
Now run the ordinary comparison on the same numbers: z = −0.47, p = 0.64. Nothing remotely significant, and in most organisations that is where the migration gets signed off as "no impact". The equivalence test says something different — a drop of a full percentage point has not been ruled out, and the interval reaches to −1.57pp, well inside the range that would have made the migration a bad trade. Neither test found a difference; only one was capable of finding sameness. The fix is sample size: this margin around a 12.8% base rate needs roughly 19,000 sessions per arm for 80% power, nearly five times what a week bought you.
A TOST p-value below α licenses one sentence: the difference in rates is smaller than the margin, at the confidence level you chose. It does not say the rates are identical, it does not transfer to a tighter margin someone else has in mind, and it does not survive being quoted without the margin attached.
The 90% interval is the better thing to circulate, because it degrades gracefully. Inside both margins, equivalence. Crossing one, not equivalent — and you can see whether you are a tenth of a point short or a mile away. Wider than the margins on both sides, and the study never had a chance, which is the usual outcome of a migration test run on a week of traffic.
Watch the two combinations that look paradoxical and are not. A result can be non-significant and non-equivalent at once, as the worked example shows; that pairing is the signature of insufficient data. It can also be significant and equivalent — with a few million sessions a 0.05pp difference has a tiny p-value and still sits deep inside a one-point margin. Reporting both tests and the margin together stops either being misread, as in reporting A/B results properly.
One caution specific to rates: equivalence on an overall conversion rate can conceal a real change in a segment that matters. If mobile fell a point and desktop rose a point, the aggregate is beautifully equivalent and something is wrong. Pre-register the two or three cuts you would act on, accept that testing several costs you multiple comparisons, and treat the rest as exploratory. Where the segments are the question, our causal inference practice can help.
In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.
Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
Migrations and re-platformings are where "no significant difference" does the most damage. We size them so the reassurance is real. Causal Inference Analysis