What is skewness?

A symmetric distribution has as much weight above its centre as below. Most business metrics do not. Revenue per user, session length, order value and time on page are all right-skewed: the bulk of observations are small, and a thin tail extends a long way to the right. Skewness is the number that quantifies that asymmetry, and it explains several things that otherwise look like separate problems.

The first consequence is that the mean stops describing a typical case. Skew pulls the mean above the median — mean revenue of £18 against a median of £4 is entirely ordinary — so "average revenue per user" describes almost nobody. That gap between mean and median is the quickest diagnostic available and needs no computation beyond what any query already returns.

The second is expense. A skewed metric has high variance relative to its mean, and required sample scales with the square of that ratio, so skewed metrics need dramatically more traffic to test on. The variance is also dominated by the tail: a handful of very large observations contribute most of it, which means the experiment's precision is effectively determined by a few dozen customers regardless of how many thousands were assigned.

The third is that the usual inference machinery takes longer to become reliable. The central limit theorem still applies, so the sampling distribution of the mean does converge to normal — but the more skewed the data, the larger the sample needed before that convergence is close enough to trust. This is where the familiar n = 30 guidance fails badly: for skewness above about 2, hundreds are needed, and for the heaviest revenue tails, thousands.

There are three standard responses and the choice depends on the question. Winsorization caps the extreme values at a pre-registered threshold, keeping the metric in currency while removing the tail's leverage. A rank-based test compares by ordering and is insensitive to magnitude entirely. Or change the metric to something naturally less skewed — conversion rather than revenue, or a capped or binarised version — accepting that it answers a slightly different question. All three are legitimate; what is not is choosing between them after seeing which gives the preferred answer.

A team examines three candidate metrics for a checkout experiment. All three measure roughly the same thing and they differ enormously in how skewed they are, which drives how much traffic each would require to detect a 5% relative effect.

Revenue per user is the metric the business cares about and by far the most expensive to test on, because it combines two sources of skew — most users contribute zero, and among those who do not, the amounts are themselves skewed. Splitting it into its two parts is what the middle row does, and it is often the better analysis: test conversion on everyone, and test order value on converters. That answers both halves at a fraction of the traffic, at the cost of needing to combine them carefully afterwards, since a change can lift conversion while lowering basket size. Note also that conversion has a skewness of 4.6 and is still cheap, which shows the coefficient alone is not the whole story — for a binary metric the variance is bounded by p(1−p), so the skew does not carry the same cost it does for an unbounded revenue tail.