The bootstrap estimates uncertainty by resampling your data with replacement thousands of times and watching how much the statistic moves. It replaces a formula for the standard error with simulation, which is what makes it work for quantities that have no convenient formula.
Every confidence interval rests on knowing how much a statistic would vary if you could repeat the experiment. For a mean there is a tidy formula for that variation. For a median, a 95th percentile, a ratio of two sums, or the difference between two conversion rates weighted by segment, there often is not — or the formula exists and depends on assumptions the data plainly violates. The bootstrap sidesteps the problem by simulating the repetition instead of deriving it.
The procedure is almost suspiciously simple. Take your sample of n observations and draw a new sample of size n from it, with replacement — so some observations appear twice or three times and others not at all. Compute your statistic on that resample. Do it ten thousand times. The spread of those ten thousand values is an estimate of how much your statistic bounces around, and the 2.5th and 97.5th percentiles of them form a 95% interval directly. No distributional assumption is made anywhere.
The logic is that your sample is the best available stand-in for the population. Drawing from it with replacement mimics drawing fresh samples from the population, and the variability you see across resamples approximates the variability you would have seen across real repetitions. That is also the source of its main limitation: if the sample is unrepresentative, the bootstrap faithfully reproduces the uncertainty of an unrepresentative sample. It quantifies sampling variability and is completely blind to selection bias.
In experimentation the cases where it earns its place are specific. Ratio metrics where the denominator is itself random — clicks per session, revenue per order — have a standard error that the naive formula gets wrong, and the alternatives are the delta method or the bootstrap. Percentile metrics such as p95 latency have no simple formula at all. And any metric where the analysis unit differs from the randomisation unit needs a cluster bootstrap, resampling whole users rather than individual events, or the interval will be far too narrow.
Two practical notes. Ten thousand resamples is a reasonable default; a thousand is enough for a standard error and too few for a stable 95% interval, since the tails are estimated from the extreme few. And the simple percentile interval is not always the best one — the bias-corrected and accelerated (BCa) variant adjusts for skew in the resampling distribution and is worth the extra computation for anything asymmetric, which most revenue metrics are.
There is no closed form, which is the point. What follows is the procedure and the two ways of turning the resampling distribution into an interval.
for b = 1..B: sample n observations with replacement, compute θ̂*ᵇB is typically 10,000. Each resample is the same size as the original, which is what preserves the sampling variability being estimated.
SE = √( Σ ( θ̂*ᵇ − θ̄* )² / ( B − 1 ) )The standard deviation of the resampled statistics. Directly comparable to an analytic standard error.
( θ̂*₍₀.₀₂₅₎ , θ̂*₍₀.₉₇₅₎ )Read the 2.5th and 97.5th percentiles off the resampling distribution. Needs B in the thousands to be stable in the tails.
resample USERS, keep all of each user's eventsRequired whenever the analysis unit is finer than the randomisation unit. Resampling events instead understates the interval, often badly.
A marketplace wants a confidence interval on the change in 95th-percentile page load time between two variants, measured over 240,000 page views from 41,000 users. There is no standard formula for the standard error of a difference in percentiles, and page views from one user are highly correlated with each other.
The variant is faster at the tail by 128 ms, with a plausible range of 54 to 201 ms once the clustering is respected.
The two intervals in the table are the finding. Resampling individual page views gives a range 3.5 times narrower than resampling users, and it is wrong — 41,000 users is the amount of independent information here, not 240,000 page views, and treating correlated events as independent manufactures precision that does not exist. A team reading the naive interval would conclude the improvement is pinned down to within ±21 ms; the honest answer is ±74 ms. This is the most common bootstrap mistake in web analytics and it is invisible, because the naive version runs happily and produces a tighter, more satisfying number. The rule is to resample at the level randomisation happened.
Learn why deviations from normality are often not a significant concern in A/B testing, understand the pros and cons of using the t-test in such scenarios, and explore alternative methods to the traditional t-test.
A/B testing analysts often encounter skewed distributions for different KPIs. Learn why outliers can be problematic, explore the challenges of identifying and handling them, and get practical guidelines for dealing with them effectively.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.