A percentile is the value below which a given share of observations falls: the 95th percentile is the point 95% of values sit under. Percentiles describe the shape of a distribution where an average collapses it into one misleading number.
A percentile locates a point in a distribution rather than summarising it. The 50th is the median, the 95th is the value 95% of observations fall below, and the 99th is where the worst one per cent begins. For anything skewed — latency, revenue, session length — a handful of percentiles describes the shape far better than a mean, which compresses everything into one number that may correspond to no actual user.
Latency is the standard case and it makes the argument concretely. Mean page load time of 800ms sounds acceptable. If the 95th percentile is 4.2 seconds, then one visit in twenty is taking over four seconds, and those are the visits people abandon. The mean cannot distinguish a system that is uniformly slightly slow from one that is fast for most people and occasionally terrible, and only the second loses customers. That is why performance work is specified in percentiles and monitored in percentiles.
The property that trips people up is that percentiles do not average. The mean of yesterday's p95 and today's p95 is not the p95 of the two days combined, because a percentile depends on the whole ordered distribution rather than on a total that can be summed. Any system reporting a p95 per hour and then averaging those to produce a daily figure is producing a number that is not a percentile of anything, and the error can be substantial when traffic varies across the day.
The same property makes them awkward in experiments. There is no simple formula for the standard error of a difference in percentiles, so the bootstrap is the standard route — resample whole users, recompute the percentile difference, and read the interval off the resampling distribution. It is more computation than a t-test and it is usually the only honest option for a percentile metric.
They are also inherently less precise than means, because a percentile depends on the observations near it rather than on all of them. Estimating p99 from ten thousand observations rests on the hundred largest, so its confidence interval is wide, and extreme percentiles are noisy enough that p99.9 on a modest sample is barely a measurement. p95 is usually the practical limit for experiment-scale data.
The definition, the two things that follow from it, and the estimator used in practice.
P( X ≤ q_p ) = pq_95 is the value 95% of observations fall below. The median is q_50.
q_p( A ∪ B ) ≠ mean( q_p(A), q_p(B) )A percentile depends on the whole ordered set. Averaging hourly p95s does not give a daily p95.
SE( q_p ) ∝ 1 / ( f(q_p) · √n )f is the density at that point. In a sparse tail the density is low, so extreme percentiles are imprecise.
cluster bootstrap the difference in percentilesNo clean analytic standard error exists — see bootstrap.
Two versions of a page are compared on load time across 180,000 sessions from 48,000 users. The means are nearly identical, and the team looks at the distribution rather than stopping there.
The means differ by 14ms and say nothing. The 95th percentile differs by 2.3 seconds, and version B is dramatically better at the tail.
This is the case percentiles exist for. Version B is slightly slower for the typical user — the median rises from 640 to 705ms — and enormously better for the users having a bad time, cutting the share of sessions over three seconds from 8.9% to 1.4%. The mean averages those two effects into 14ms and reports nothing at all. Which version to ship depends on what you believe about the relationship between load time and abandonment, and the usual answer is that the tail matters more, since a 65ms difference is imperceptible and a four-second wait is where people leave. Two technical notes. The interval comes from a bootstrap clustered by user, because 180,000 sessions from 48,000 users are not independent, and treating them as such would have given an interval roughly half as wide. And p99 is reported here as descriptive context rather than as a tested quantity — at this sample size its interval would be wide enough that the difference, while clearly large, would not be tightly estimated.
Learn why deviations from normality are often not a significant concern in A/B testing, understand the pros and cons of using the t-test in such scenarios, and explore alternative methods to the traditional t-test.
A/B testing analysts often encounter skewed distributions for different KPIs. Learn why outliers can be problematic, explore the challenges of identifying and handling them, and get practical guidelines for dealing with them effectively.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.