The Mann-Whitney U test compares two groups by ranking every observation and asking whether one group's values tend to sit higher in the ordering. Because it uses only the ranks, a single enormous value counts as one place rather than as its magnitude.
UPool every observation from both groups, sort them, and give each one its position in that ordering. If the two groups are alike, their ranks should be thoroughly mixed; if one group tends to produce larger values, its members will cluster towards the top. The Mann-Whitney U test measures how far the actual mixing is from what chance would produce. Nothing about the magnitudes survives this step — the largest value in the dataset is simply the highest rank, whether it exceeds the runner-up by a penny or by fifty thousand pounds.
That insensitivity is the point. Revenue per user, order value, session duration and lifetime value all share a shape: most observations modest, a long right tail, and occasionally a customer who spends more than the next hundred combined. A mean is a poor summary of that distribution and a t-test on it can be dominated by two or three people. Ranking flattens the tail and produces a comparison that does not hinge on whether one whale happened to land in the treatment arm.
The cost is that it answers a different question, and the difference matters more than it first appears. A t-test asks whether the group *means* differ. Mann-Whitney asks whether a randomly chosen observation from one group tends to exceed one from the other — a claim about the typical case rather than the total. For a product decision those can diverge sharply: a pricing change that slightly reduces what most customers spend while substantially increasing what a few spend can be a clear win on revenue and a clear loss on ranks. The test is not wrong there; it is answering the question you asked rather than the one you needed.
It is also not assumption-free, despite the "non-parametric" label. To read a significant result as a shift in the middle of the distribution, the two distributions need roughly the same shape — if the treatment changes the spread rather than the location, U can be significant with identical medians. And it is not automatically less powerful than a t-test: on heavily skewed data it is frequently more powerful, because it is not spending precision on outliers. On symmetric data it retains about 95% of the t-test's power, which is a small premium for robustness.
In practice the choice is usually decided by a quick check rather than a rule: does removing the top 0.1% of observations change the conclusion? If it does, the mean is being carried by a handful of points and either rank-based testing or winsorising is called for. If it does not, use the t-test and report a difference in units the business recognises. We work through the wider question in what to do when data is not normally distributed.
The statistic counts how often one group beats the other across all possible pairings. At experiment sample sizes it is read through a normal approximation rather than an exact enumeration.
U₁ = R₁ − n₁(n₁ + 1) / 2R₁ is the sum of ranks in group 1. Subtracting the minimum possible rank sum leaves how far above the floor the group landed.
U = number of pairs (i, j) where xᵢ > yⱼEquivalent to the formula above and much more interpretable: out of every cross-group pairing, how often did group 1 win?
P( X > Y ) = U / ( n₁ · n₂ )The common-language effect size. 0.5 means indistinguishable; 0.62 means a random treated user beats a random control 62% of the time.
z = ( U − n₁n₂/2 ) / √( n₁n₂(n₁ + n₂ + 1) / 12 )Used above roughly 20 per group, which every online experiment satisfies — see the Wilcoxon calculator.
A marketplace tests a checkout change on revenue per session, 6,000 sessions per arm. Control averages £41.10 and the variant £44.60, an 8.5% lift that looks convincing. Both arms are heavily right-skewed: the median in each is close to £22, and the top 0.1% of sessions carry over a fifth of total revenue.
The t-test says nothing detectable; the rank test says the variant is ahead. The two disagree, and the reason is visible in the last row.
The £3.50 gap in means is largely six sessions. Remove them and the t-test collapses to p = 0.31, which tells you the mean comparison was never resting on the other 11,994 observations. The rank test, unable to see magnitudes, finds a modest but consistent shift: a random variant session beats a random control session 52% of the time, up from the 50% of no effect. Both readings are true and they support different decisions. If the business case is total revenue, six sessions is far too thin a basis to ship on and the honest answer is that this test is underpowered for a skewed metric — cap the outliers and rerun, or plan for more traffic. If the case is that the checkout works better for the typical customer, the rank test supports it. What would be indefensible is running both and reporting whichever crossed 0.05.
Learn why deviations from normality are often not a significant concern in A/B testing, understand the pros and cons of using the t-test in such scenarios, and explore alternative methods to the traditional t-test.
A/B testing analysts often encounter skewed distributions for different KPIs. Learn why outliers can be problematic, explore the challenges of identifying and handling them, and get practical guidelines for dealing with them effectively.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.