A confidence interval is a range of values that is consistent with your data, computed so that intervals built this way capture the true value a stated share of the time — usually 95%. It reports how precisely you have measured something, which a single estimate cannot.
CIAny measurement taken from a sample is wrong by some amount, and the interesting question is by how much. A confidence interval answers that. Instead of reporting "the new flow lifted conversion by 7.4%", it reports "by somewhere between 1.4% and 13.5%" — and that second statement is the one a business can act on, because it makes the uncertainty visible instead of burying it behind a decimal point.
The 95% is a statement about the *procedure*, not about the particular interval on your screen. If you repeated the experiment many times and built an interval each time the same way, about 95 of every 100 would contain the true value. Any single interval either does or does not; there is no probability left in it once the data are in. This is the distinction people find least intuitive and it has a practical payoff: it is why you cannot say "there is a 95% chance the lift is between 1.4% and 13.5%", and why a Bayesian credible interval — which does support that sentence — is a different object with different assumptions.
Three quantities set the width, and they are the same three that set statistical power. Variability in the outcome widens it. Sample size narrows it, at the square root — four times the users to halve the width. And the confidence level trades width for coverage: a 99% interval is about 32% wider than a 95% one. Nothing else is available. An interval that is too wide to support a decision is telling you the experiment was too small, and no amount of re-analysis will fix that.
The relationship to significance testing is exact rather than approximate. A 95% interval on a difference excludes zero precisely when the two-sided p-value is below 0.05 — they are the same arithmetic presented two ways. Which is why reporting the interval strictly dominates reporting the p-value: it carries the significance verdict for free and adds the magnitude, which the p-value throws away. Our argument for making that the house standard is in how to properly report A/B testing results.
The habit worth building is to read the interval against a threshold rather than against zero. Draw the smallest effect you would ship — the minimum detectable effect you designed against — and see where it falls. An interval entirely above it is a ship decision. Entirely below it is a genuine no, and a much stronger statement than a non-significant p-value. Straddling it means the experiment did not resolve the question, which is a third outcome most reporting templates have no room for and most decisions badly need.
Almost every interval you will meet has the same shape: the estimate, plus and minus a critical value times its standard error. What changes between methods is which distribution supplies the critical value, and whether the symmetric form is valid at all.
CI = estimate ± z₁₋α/₂ · SE(estimate)z₁₋α/₂ = 1.960 for 95%, 1.645 for 90%, 2.576 for 99%. Small samples of a mean use the t quantile on n − 1 degrees of freedom instead.
(p̂₁ − p̂₂) ± z₁₋α/₂ · √( p̂₁(1−p̂₁)/n₁ + p̂₂(1−p̂₂)/n₂ )The standard A/B testing interval on an absolute difference in conversion rates.
(x̄₁ − x̄₂) ± t₁₋α/₂,ν · √( s₁²/n₁ + s₂²/n₂ )ν is the Welch-Satterthwaite degrees of freedom. Welch rather than Student because equal variances are an assumption you rarely have grounds for — see the two-sample t-test calculator.
( p̂ + z²/2n ± z·√( p̂(1−p̂)/n + z²/4n² ) ) / ( 1 + z²/n )Not the textbook p̂ ± z·√(p̂(1−p̂)/n), which fails badly for rare events and can run below zero. The proportion confidence interval calculator uses Wilson for that reason.
The checkout test from earlier: control converts 2,150 of 50,000 sessions (4.30%), variant 2,310 of 50,000 (4.62%). The p-value was 0.0147, so the result is significant. Product said before the test that they would only ship a lift of 5% relative or more, which at this baseline means 0.215 percentage points.
95% CI: 0.06 to 0.58 percentage points absolute, which is 1.4% to 13.5% relative. The interval excludes zero, matching p = 0.0147.
The interval crosses the ship threshold rather than clearing it. Everything from a 1.4% lift — well below what product said was worth shipping — to a 13.5% one is consistent with this data. So the honest summary is not "the variant won" but "the variant is better, and we do not yet know whether it is better enough." That is a materially different conversation, and it is invisible in a report that says only p = 0.015. Doubling the sample would roughly halve the width and resolve it; shipping on the point estimate risks banking a 7.4% lift that turns out to be 2%.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.