Paired t-test calculator

Use a paired t-test when every observation in one condition has a natural partner in the other: the same user before and after a change, the same store in two consecutive periods, two units matched on the covariates that drive the outcome. The test does not really compare two groups — it collapses each pair to one difference and asks whether their average is zero. Hence n is the number of pairs, not of measurements, and the SD that governs the sample size is the SD of the differences, not of the outcome.

The whole argument for pairing lives in the correlation between the two measurements. Because σ_d = √(σ₁² + σ₂² − 2rσ₁σ₂), a strong positive correlation shrinks the noise you test against: with equal SDs and r = 0.8 the difference SD is 0.63σ, and the design needs roughly a fifth of the measurements an independent-groups test would. At r = 0 it needs exactly the same measurements and has half the degrees of freedom — pairing has cost you something and bought you nothing. At r < 0 it is actively worse. This is the lever CUPED pulls, and one of the ways to buy power without buying traffic.

If the units in your two conditions are different people the pairing does not exist, and you want the two-sample t-test: pairing observations that were never paired is not a conservative mistake, it is the wrong standard error. Paired binary outcomes want McNemar's test, which runs on the same idea and discards the pairs that agree. Heavily skewed differences want the Wilcoxon signed-rank test. Three or more conditions on the same unit need a repeated-measures model, not one-way ANOVA.

The paired test trades one assumption set for another. It drops independence between groups entirely — the two measurements are meant to be correlated — and takes on these instead:

• The pairs are independent of each other. Within a pair correlation is the point; across pairs it is a violation — two stores in the same shopping centre are not two independent pairs.
• The pairing is fixed by the design, not chosen after the data arrived. Matching units post hoc on a variable you have already inspected, or dropping pairs whose difference looks wrong, leaves the test with no honest error rate.
• The differences are approximately normal — not the raw measurements. This is the assumption people misread most often: two wildly skewed distributions can produce symmetric differences, and two well-behaved series can produce differences with a heavy tail. Plot the differences, not the levels. Above about 30 pairs the Central Limit Theorem does most of the work; below that see what to do when data is not normally distributed.
• No single pair dominates. The difference SD comes from the same n differences as the mean, so one extreme pair either inflates the denominator or, running with the effect, manufactures one — see tackling outliers.
• None of this rescues a before-and-after design from the absence of a control. The test says the mean changed, not that you changed it: seasonality, a pricing move and a competitor's campaign all sit in the same window. There, a geo test or an explicit causal design is what you need.

You are rolling out a new shelf layout, measuring each store for four weeks before and four weeks after. Revenue per store per period has a standard deviation across stores of about $8,000, and consecutive periods have historically correlated at around 0.85. The smallest lift that would justify the rollout is $1,500 per store.

The comparison worth making is against the unpaired version: detecting a $1,500 difference between two independent sets of stores with an $8,000 SD would need something in the region of 450 stores per arm, close to 900 in all. Pairing does not make the effect bigger — it removes the stable between-store differences (location, footfall, catchment income) that are most of that $8,000. Note also how fragile the answer is: at r = 0.7 rather than 0.85, σ_d rises to roughly $6,200 and the requirement roughly doubles. The correlation is the input to interrogate, not the effect size.

Read the confidence interval on the mean difference before the p-value. It is in the units you care about and gives the range of true differences the data is compatible with. An interval of [$120, $3,900] is significant and nearly useless: it cannot distinguish a rounding error from a result that funds the programme twice over. An interval of [−$200, $180] is not significant but is genuinely informative, because it rules out anything worth acting on — the argument of our note on reporting results properly.

Then check the realised s_d against the one you assumed at design time — that comparison is the post-mortem for the whole study. If it came in well above your assumption, the two measurements were less correlated than history suggested and the test is underpowered for reasons unrelated to the effect. That is a different diagnosis from "there is no effect", and it points to a better matching variable rather than to abandoning the idea.

The p-value carries a smaller claim than most write-ups give it: the probability of a mean difference this large if the true difference were zero. Nothing about causation, and in a before-and-after design nothing separating your intervention from everything else that happened in between. Where the two periods are not comparable, add a concurrent control — which turns this into a two-sample comparison on the differences — or build the counterfactual with causal inference.

Finally, a non-significant paired test does not show the two conditions are interchangeable. If the question is whether the cheaper supplier hurts anything, nominate a margin in advance and run an equivalence test, or a non-inferiority test when only one direction of harm matters. Those are claims you can put in a decision memo; a large p-value is not.