A/B test sample size calculator

Use this when the outcome is binary — converted or not, signed up or not, still active on day 7 — and two independent groups were formed by random assignment. That is most of what gets called an A/B test. The sample-size tab answers the question worth asking before launch, which is not "how long shall we run it?" but "how small a lift must we be able to see, and what does that cost?"

Neighbouring situations need a different instrument. One group measured against a fixed target — an SLA, a contractual floor — is a one-proportion z-test. Counts small enough that a cell holds single digits belong in Fisher's exact test, where the p-value is computed rather than approximated. Three or more variants is a chi-square test for the overall question, then targeted pairwise comparisons with a correction — running every pair unguarded is how a four-arm test manufactures a winner, as the problem of multiple comparisons sets out. Same people measured twice is paired data, and wants McNemar's test.

The other mismatch is the question itself. This test detects a difference; it cannot demonstrate the absence of one. To show a cheaper vendor or a simplified flow performs no worse, use an equivalence test or, when one direction worries you, a non-inferiority test. And if what matters is how long users take to convert rather than whether they do, the binary summary discards timing that the log-rank test keeps.

Five things have to hold. The first two are load-bearing; the rest are usually satisfied by any competently run experiment.

• Random assignment, with one observation per randomised unit. If assignment leaked, no amount of statistics repairs it. Check the split against the ratio you intended before you look at the metric: sample ratio mismatch is the cheapest bug detector you have, and the basics of randomisation repay a re-read before launch.
• The unit of analysis matches the unit of randomisation. Bucket by user but count sessions and the same user contributes several correlated rows, so the standard error is understated and the p-value too small. Aggregate first, or model the clustering.
• The sample size was fixed in advance. Checking daily and stopping at the first p below 0.05 pushes the real false-positive rate far above 5%. To keep the option of stopping early, adopt a sequential design that budgets for the looks.
• The normal approximation is adequate — comfortably so once each of the four cells expects ten or more. Below that its p-values are untrustworthy at the third decimal; switch to the exact test. This is the assumption most calculators fail silently on, since they will happily return a p-value for a 3-versus-1 table.
• One primary metric, nominated in advance. Five metrics at 5% each gives roughly a one-in-four chance of a false positive somewhere — see multiple comparison corrections.

Your checkout converts at 4.0%. Product will not ship for less than half a percentage point, so 4.5% is the smallest result worth detecting. You want 80% power at the conventional 5% two-sided level, an even split, and roughly 6,000 eligible visitors reach the page each day.

Nine days buys a four-in-five chance of detecting a lift of half a point, if that lift is really there. It buys much less against a smaller true effect: at a real lift of a quarter of a point this design has about 30% power, so the likeliest outcome is a non-significant result written up as "no impact" when it was really "no ability to tell". Sizing for that quarter-point needs just under 100,000 per arm — four times the traffic for half the sensitivity, which is the trade every experiment roadmap is really negotiating.

Start with the confidence interval on the difference, not the p-value. Both settle the yes/no question — the interval excludes zero exactly when p is below α — but only the interval says which lifts the data is consistent with. An observed +0.6 points with an interval of [+0.05, +1.15] is significant and almost uninformative; +0.1 points with an interval of [−0.15, +0.35] is not significant and genuinely useful, because it rules out anything you would have acted on. That is the core recommendation in reporting A/B test results properly.

The p-value is the probability of a difference at least this large if both variants truly converted at the same rate. It is not the probability that the variant is better, not the probability the result replicates, and 0.049 and 0.051 are the same evidence.

Be careful with relative lift, which is what stakeholders remember. Its interval is asymmetric and wider than the point estimate suggests, and the estimate is biased upward among tests that reached significance — a winner is likelier to have been helped by noise than hurt by it. That is much of why the uplift in an A/B test often differs from what shows up after rollout.

For a bound on one arm's rate rather than the gap between two, the proportion confidence interval calculator does that properly. And if the interval is too wide to support a decision, the honest conclusion is that the test was underpowered, not that the change does nothing — we design experiments where that is settled before launch.