What is a sampling distribution?

There are two distributions in any analysis and confusing them causes most of the trouble. The first is the distribution of your data: how revenue is spread across users, which for revenue is skewed with a long right tail. The second is the sampling distribution: how the *average* of that revenue would vary if you drew a fresh sample of the same size, over and over. Everything inferential — the standard error, the p-value, the confidence interval — is computed from the second.

This is the rung that makes the rest of the vocabulary coherent. The standard error is simply the standard deviation of the sampling distribution. A confidence interval is a range read off it. A p-value is a tail area under it. None of those quantities is about your data directly; they are about how much the number you computed from your data would move if you did it again.

The reason it is usable at all is the central limit theorem, which says that for a mean the sampling distribution tends towards normal as the sample grows, regardless of the shape underneath. That is what allows a t-test on revenue data that looks nothing like a bell curve. The normality assumption people worry about attaches to this distribution, not to the raw values, and knowing which one is being assumed resolves most of the anxiety about it.

You never observe it, which is what makes it feel abstract. You have one sample, not a thousand, so the sampling distribution is inferred rather than seen — usually from theory, using the formula for the standard error. The bootstrap is the alternative: resample your own data thousands of times and watch the statistic move, which builds an empirical approximation of exactly this object. That is worth knowing because it makes the concept concrete rather than theoretical.

The practical payoff is knowing when the machinery breaks. The theory assumes independent observations, so correlated data — several sessions from one user — makes the real sampling distribution wider than the formula predicts, and every interval too narrow. And for a badly skewed metric at small sample sizes the convergence to normality has not happened yet, so the formula's answer is wrong in a way no amount of care about the raw data would reveal.

A dataset of 40,000 users' monthly revenue is heavily right-skewed: median £4, mean £18.40, and a small number of users above £2,000. A simulation repeatedly draws samples of three different sizes and records the mean each time, to show what the sampling distribution looks like at each.

The first and last rows together are the whole point. The underlying data stays skewed no matter how much of it you collect — that is a property of the population and does not change. What changes is the distribution of the average, which is normal enough for standard inference well before n = 1,000 and clearly is not at n = 30. So the familiar advice that a t-test needs normally distributed data is asking about the wrong distribution, and the equally familiar 'n = 30 is enough' is asking about the right one and giving a number that is far too small for a metric this skewed. The honest guidance is that the required sample depends on how skewed the data is, and that skewness of 6.8 needs hundreds rather than dozens. A bootstrap gives the answer directly for any specific dataset without needing the rule of thumb at all.