What is the normal distribution?

The normal distribution is the symmetric bell curve, defined entirely by its mean and its standard deviation. It is symmetric about the mean, has thin tails, and follows the familiar rule that about 68% of values fall within one standard deviation, 95% within two and 99.7% within three. Its prominence is not because natural quantities are usually normal — many are not — but because of what happens to averages.

That is the central limit theorem, and it is the single most useful fact in applied statistics. Averages of many independent observations tend towards a normal distribution regardless of the shape of the original data. So a t-test does not require your data to be normal; it requires the *sampling distribution of the mean* to be approximately normal, which the central limit theorem usually delivers at experiment-scale sample sizes even when the raw data are wildly skewed.

This resolves the anxiety that surrounds normality testing in practice. Revenue per user is heavily right-skewed — most users spend nothing, a few spend a great deal — and a normality test on it will reject decisively at any reasonable sample size. That rejection is almost always irrelevant. With tens of thousands of observations per arm, the mean is close enough to normal for a t-test to behave correctly, and formal normality tests are actively unhelpful here: they reject trivial departures in large samples and lack the power to detect real ones in small samples, which is precisely backwards.

Where non-normality does bite is worth knowing precisely. Small samples, where the central limit theorem has not had enough observations to work with — under about thirty, and more for heavily skewed data. Extreme skew combined with a heavy tail, where the mean itself is dominated by a few outliers and estimates become unstable between samples. Statistics that are not averages, such as a maximum or a ratio of two random quantities, which the theorem says nothing about. And prediction intervals for individual values rather than for a mean, which do depend on the shape of the raw data.

When it genuinely is a problem, the responses are ordered by how much they change the question. Trim or winsorise the tail, which keeps the metric interpretable. Transform — a log is standard for right-skewed revenue — while remembering that you are now testing a difference in log means, which is a different quantity. Use a rank-based test such as Wilcoxon, which drops the normality assumption and tests a shifted distribution rather than a mean. Or bootstrap the interval, which makes no distributional assumption at all and is the cleanest option when compute is not a constraint. We cover the practical sequence in what to do when data is not normally distributed.

Revenue per user in an experiment: 71% of users spend nothing, the mean is £6.80, the median is £0, and the standard deviation is £91. A Shapiro-Wilk test rejects normality at p < 0.0001. An analyst asks whether a t-test is valid at 40,000 users per arm.

The normality test is answering a question nobody asked. It is testing whether the raw revenue values are normal — they obviously are not, with 71% zeros and a skewness of 14 — while the t-test only requires the *difference in means* to be approximately normal, and at 40,000 users per arm the central limit theorem has delivered that comfortably. The bootstrap is the check that settles it: it makes no distributional assumption whatsoever, and it agrees. Where this would break down is at a few hundred users per arm, or if a single £40,000 order were driving the mean — so the diagnostic worth running is not a normality test but a look at how much the estimate moves when the largest few observations are removed.