Winsorization replaces values above a chosen threshold with the threshold itself, so extreme observations are pulled in rather than removed. It cuts the variance a heavy tail contributes while keeping every user in the analysis.
Revenue metrics have a shape that makes them difficult: most users contribute nothing or very little, and a handful contribute enormously. One customer spending £40,000 can shift a group mean further than the treatment did, so which arm they happened to land in decides the experiment. Winsorization addresses this by capping — every value above the 99th percentile, say, is replaced by the value at the 99th percentile. The extreme users stay in the analysis and stop dominating it.
The distinction from trimming matters. Trimming deletes the extreme observations entirely, which changes the population being analysed and breaks the randomisation if the number deleted differs between arms. Winsorization keeps every unit and only compresses their values, so the two arms still contain the same people they were assigned. That is why it is the safer default for experiment analysis, and why the terms should not be used interchangeably despite often being treated as synonyms.
The variance reduction is typically large because variance weights squared deviations — an observation ten times the mean contributes a hundred times as much as one at the mean. Capping at the 99th percentile affects one per cent of users and routinely removes 40 to 60 per cent of the variance of a revenue metric, which by the quadratic sample-size relationship means less than half the traffic for the same minimum detectable effect. Few interventions in experimentation are that cheap.
The single rule that makes it legitimate is that the threshold is fixed before the results are seen. A cap chosen after looking at the data is not a statistical method; it is a choice about which answer to obtain, and with a heavy-tailed metric the answer genuinely does move as the threshold slides. Pick the percentile from historical data, write it into the analysis plan alongside the sample size calculation, and apply it identically to both arms.
What it costs is honesty about the question. A winsorized revenue metric is no longer total revenue — it is revenue with the top tail compressed, and a treatment whose entire effect is on whales will be invisible under it. For most product changes that is an acceptable trade, because a result that depends on three customers was never reliable evidence anyway. When the tail genuinely is the business, the alternatives are a rank-based test such as Mann-Whitney or accepting that the metric needs far more traffic.
One substitution rule, plus the variance identity that explains why capping so few observations changes the answer so much.
x' = min( x, q_p )q_p is the value at percentile p, computed on the pooled data across arms so the cap is identical in both.
x' = min( max( x, q_low ), q_high )Rarely needed for revenue, which has a floor at zero. Useful for metrics like latency deltas that run both ways.
Var = E[ ( x − μ )² ]Squared deviations. A value 10× the mean contributes 100× as much variance as one at the mean.
n ∝ σ², so halving variance halves required trafficCapping at p99 typically removes 40-60% of a revenue metric's variance — see the two-sample t-test calculator.
A marketplace analyses a checkout test on revenue per user across 120,000 users per arm. The raw metric is dominated by a small number of very large orders, and the team wants to know what capping at various percentiles would do — and, critically, they compute this on last quarter's data before the experiment reads out.
Capping at the 99th percentile touches 1% of users, halves the standard deviation, and cuts the required sample by roughly a factor of four.
The last row is the one that requires a decision rather than a calculation. Those 1,200 users carry 22% of all revenue, so capping them is not a cosmetic adjustment — it materially changes what the metric measures, and a treatment that works mainly by increasing large orders would be invisible under a p99 cap. For a checkout usability change that is almost certainly fine, since such a change should affect the ordinary customer and any result resting on a handful of whales would not have been trustworthy. For a test of a new enterprise pricing tier it would be exactly wrong. The p95 column is included to show where this stops being reasonable: capping 5% of users compresses too much of the real distribution to still be called revenue. The defensible choice here is p99, decided now, applied to both arms, and reported explicitly as a capped metric.
A/B testing analysts often encounter skewed distributions for different KPIs. Learn why outliers can be problematic, explore the challenges of identifying and handling them, and get practical guidelines for dealing with them effectively.
Learn why deviations from normality are often not a significant concern in A/B testing, understand the pros and cons of using the t-test in such scenarios, and explore alternative methods to the traditional t-test.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.