Two-sample t-test calculator

Use a two-sample t-test when you have one continuous measurement and two independent groups, and you want to know whether their averages differ by more than sampling noise would explain. Revenue per user in control against treatment, session length on two onboarding flows, delivery time from two warehouses — anything where each observation belongs to exactly one group and no observation is paired with another.

The independence requirement is the one that quietly fails in product work. If the same user appears in both arms, or the same user contributes several sessions, the observations are not independent and the test will report a p-value smaller than the evidence deserves. Repeated measurements on the same unit want the paired t-test or a model that accounts for the clustering; three or more groups want one-way ANOVA rather than a series of pairwise t-tests, which inflates the false-positive rate — see the problem of multiple comparisons.

If your outcome is a rate rather than a measurement — conversion, click-through, retention at day 7 — you want the A/B test sample size calculator instead. And if your question is "are these two groups the same?" rather than "do they differ?", a non-significant t-test does not answer it; an equivalence test does.

The t-test rests on three assumptions, and they are not equally fragile:

• Independence, within and between groups. This is the assumption that matters. There is no correction to apply after the fact and no diagnostic plot that reveals it — it is a property of how you assigned units, not of the data you collected.
• Approximate normality of the group means, not of the raw data. The Central Limit Theorem does most of the work here: with a few hundred observations per group, even visibly skewed data produces means that are near enough normal. Small samples of heavily skewed or long-tailed data are the real problem, and revenue per user is exactly that shape. See our note on what to do when data is not normally distributed, or use the Wilcoxon test if you would rather not assume it.
• Equal variances — for Student's version only. Welch's version drops this assumption at a cost of a fraction of a degree of freedom, which is why it is the default here and the default in R. There is no good reason to run Student's t-test after a variance test; that two-step procedure has worse error rates than simply using Welch throughout.
• Outliers are not an assumption, but they will dominate the result. A single extreme value can move both the mean and the standard deviation enough to flip a conclusion — see tackling outliers in A/B testing.

You are testing a new checkout flow and the outcome is revenue per session. Historical sessions average $10.00 with a standard deviation of $5.00, and the smallest improvement that would justify shipping the change is $2.00. You want 80% power at the conventional 5% two-sided significance level, with traffic split evenly.

Two hundred sessions is the cost of being able to detect a $2.00 lift four times out of five. It is not the number that guarantees you will see one: if the true lift is $1.00 rather than $2.00, this design has about 29% power, and the most likely outcome is an inconclusive test that gets read as "no effect". Halving the detectable difference roughly quadruples the sample size, which is the single most useful thing to know before negotiating a test's scope.

The p-value is the probability of seeing a difference at least this large if the two groups genuinely had the same mean. It is not the probability that the null hypothesis is true, and it is not the probability that the result will replicate. A p of 0.03 says the data would be somewhat surprising under no effect; it says nothing about how large the effect is.

Read the confidence interval first. It carries everything the p-value carries — it excludes zero exactly when p < α — and it also tells you the range of differences the data is consistent with. An interval of [$0.10, $4.20] is technically significant and practically useless: it cannot distinguish a trivial lift from a transformative one. An interval of [−$0.20, $0.30] is not significant, but it is genuinely informative, because it rules out anything worth acting on. Reporting the interval rather than the p-value alone is the main recommendation in our guide to reporting A/B test results properly.

A non-significant result is not evidence of no difference. If you need to demonstrate that two options are interchangeable — that a cheaper vendor performs no worse, that a refactor changed nothing — the t-test is the wrong instrument. Use an equivalence test to show the difference falls inside a margin you nominate in advance, or a non-inferiority test if only one direction concerns you.

Finally, the power you calculated before the test tells you nothing new after it. Post-hoc power computed from the observed effect is a deterministic function of the p-value and adds no information; if the test was inconclusive, the useful question is what the confidence interval rules out, not what the power "was".