What is a t-test?

Two groups produce two averages, and those averages are never identical. The question a t-test answers is whether the gap between them is bigger than the gap you would expect from random assignment alone. It answers it by putting the difference on a scale set by the data's own variability: how many standard errors apart are these two means? That ratio is the t statistic, and everything else is looking up how often a value that extreme occurs when nothing is going on.

The scaling is the whole idea, and it is why a t-test cannot be replaced by eyeballing the difference. A £4 gap in average order value is enormous if orders cluster tightly around £30 and invisible if they range from £5 to £900. Dividing by the standard error makes those two situations comparable, which is what lets one table of critical values serve every experiment ever run. The resulting number is compared against the t-distribution rather than the normal curve, because the spread was estimated from the same small sample rather than known in advance.

There are three tests sharing the name and they are not interchangeable. Student's assumes both groups have the same variance; Welch's does not, and is the correct default for online experiments, where a treatment that changes behaviour usually changes its spread as well. The paired t-test is a different design entirely — it applies when each observation in one group has a partner in the other. Most statistical software still defaults to Student's, which is the single most common way a correct-looking analysis is quietly using the wrong test.

The assumption people worry about is normality, and it is mostly the wrong worry. The test needs the *sampling distribution of the mean* to be roughly normal, not the data — and the central limit theorem delivers that for any reasonable sample size on most real distributions. The familiar "n = 30" threshold is folklore: for mildly skewed data thirty is generous, and for revenue data with a long right tail and a handful of enormous customers, thirty is nowhere near enough and several thousand may not be either. The honest check is whether a few extreme values dominate the mean, and when they do the answer is capping them or switching to a rank-based test.

What the test does not tell you is how big the difference is or whether it matters, and reading a small p as a large effect is the error this whole section exists to prevent. Report the difference in the units the business uses alongside a confidence interval — a t-test that clears significance on a gap of £0.04 has told you something true and useless. We make the longer version of that argument in how to report A/B test results.

A checkout redesign is tested on average order value. Control shows a mean of £48.20 with a standard deviation of £31.40 across 4,100 orders; the variant shows £50.05 with a standard deviation of £34.90 across 4,050 orders. The spread is visibly different between the arms, which already rules out the pooled version.

Significant, and the interval is what makes it actionable. A gap this large turns up by chance about once in eighty identical experiments where the two checkouts were genuinely equivalent, so the difference is real. But the plausible range runs from £0.41 to £3.29, and those two ends imply very different business cases — at the bottom the redesign barely pays for the engineering time, at the top it is one of the best changes of the year. Two details are worth noticing. The variances differ by more than 10%, so using Student's pooled test here would have quietly overstated the precision. And order value is right-skewed, so it is worth confirming that no single enormous order is driving the £1.85; if one is, the mean is the wrong summary and the conclusion should not survive removing it.