What is an outlier?

An outlier is an observation that sits far from the others. In business data they are usually real: the customer who bought forty units for an office, the account that placed a £40,000 order, the session that lasted nine hours because someone left a tab open. Some are genuine errors — a test transaction, a bot, a duplicated event, a currency recorded in the wrong unit — and separating those two categories is the whole of the problem, because the correct treatment is entirely different.

The reason they matter so much in experimentation is arithmetic. The mean and the variance are both computed from squared deviations, so a single extreme value moves the variance far more than it moves the mean — a value ten standard deviations out contributes a hundred times what a typical one does. On a revenue metric with a long tail, a handful of orders can supply most of the variance, which means they largely determine the width of your confidence interval and therefore whether the test concludes anything at all.

They also break the comparison in a subtler way. If one arm happens to receive the £40,000 order and the other does not, the difference in means reflects that one customer rather than the treatment. This is not a bias — randomisation still holds — but it is enormous variance, and it means a result can flip on a single row. Checking whether the conclusion survives removing the largest few observations is one of the most informative diagnostics available, and it takes a minute.

The rule that keeps this honest is that the handling must be decided before the data are seen. Winsorising at the 99th percentile — replacing values above it with the percentile itself — is the usual choice for revenue metrics, because it keeps every user in the analysis while capping their influence. Trimming removes them entirely, which discards information and changes the population being described. Analysing a log or a rank-based statistic sidesteps the issue by changing the question. All of these are defensible; deciding among them after seeing which one produces significance is not.

Genuine errors are a different matter and should be excluded — but the exclusion rule must be blind to the arm. A filter that drops sessions under five seconds will fire more often on a slower variant, which reintroduces selection bias into a randomised design and can be detected as a sample ratio mismatch. Define exclusions on pre-assignment characteristics where possible, apply them identically to every arm, and check the arm counts afterwards. We work through the practical sequence in tackling outliers in A/B testing.

An experiment on revenue per user, 60,000 per arm. The variant shows +£0.42 per user, p = 0.03. The analyst checks robustness by recomputing after capping the top 0.1% of orders, and separately after removing the single largest order.

The reported win is one customer. That customer is real and their order is real, but a result that flips on a single row is not a finding about the treatment — it is a finding about which arm happened to receive an unusual purchase, which randomisation makes a coin flip. The uncomfortable part is that the correct action was not available at this point: the winsorising rule should have been in the design document, because choosing it now means choosing between +£0.42 significant and +£0.09 not, with the results already visible. The defensible write-up reports both, states that the effect is not robust to a single observation, and commits to winsorising at the 99th percentile for every revenue test going forward.