Your test has finished. Enter the conversions and the observations in each group, and get the effect, its confidence interval and a p-value — with the interval first, because it is the one that says which effects your data is actually consistent with. Planning a test rather than reading one? The sample size calculator is the other half of this page.
Enter your numbers on the left and the result appears here.
Use this when an experiment has finished, the outcome is binary — converted or not, signed up or not, still active on day 7 — and two independent groups were formed by random assignment. Four numbers describe the whole result: how many people entered each group, and how many of them converted.
Before you read any of it, check the split. If assignment was meant to be even and the two group sizes differ by more than chance explains, the metric is not worth interpreting yet — a sample ratio mismatch means something upstream is broken, and no amount of statistics downstream repairs it. The chi-square goodness-of-fit test is how you check it.
Neighbouring situations want a different page. Counts small enough that a cell holds single digits belong in Fisher's exact test, where the p-value is computed rather than approximated. One group against a fixed target is a one-proportion z-test. The same people measured twice is paired data and wants McNemar's test. And if the question is "are these two the same?" rather than "do they differ?", no significance test answers it — an equivalence test does.
The analysis inherits its assumptions from the design. Three of them can still be checked after the fact, and one cannot be repaired at all.
The p-value is a pooled z-test and the interval is not, which is deliberate rather than an inconsistency: the test is evaluated under the null that both groups share one rate, and the interval is not conditioned on that null being true. Reporting a pooled interval alongside a pooled test would be internally tidy and would give an interval that does not match the estimate above it.
z = (p̂₂ − p̂₁) / √( p̄(1−p̄)(1/n₁ + 1/n₂) ), p̄ = (x₁ + x₂)/(n₁ + n₂)Pooling is right here: the statistic is evaluated under the null that both groups convert at one rate, so that rate is estimated from everything. Squaring this z gives exactly the Pearson chi-square statistic for the same 2×2 table, which is why the two never disagree.
(p̂₂ − p̂₁) − √((p̂₂ − l₂)² + (u₁ − p̂₁)²) to (p̂₂ − p̂₁) + √((u₂ − p̂₂)² + (p̂₁ − l₁)²)Built from each group's Wilson limits (lᵢ, uᵢ) rather than one Wald standard error. It stays inside [−1, 1], holds close to nominal coverage at low conversion rates, and does not collapse to zero width when a group records no events — all three of which Wald gets wrong exactly where conversion tests live.
exp( ln(p̂₂/p̂₁) ± z·√( (1−p̂₁)/x₁ + (1−p̂₂)/x₂ ) ) − 1A separate interval, not a rescaling of the one above. Dividing the difference interval by the observed control rate — what most dashboards do — treats an estimate as a known constant and comes out too narrow, badly so when the control group is small. Undefined when either group converted nobody, and reported as such rather than filled in.
z′ = ( |p̂₂ − p̂₁| − ½(1/n₁ + 1/n₂) ) / √( p̄(1−p̄)(1/n₁ + 1/n₂) )Yates' correction for a 2 × 2 table: shrink the observed difference by half a discrete step before dividing, which makes the p-value larger. Say whether you applied it, because on a borderline result it moves p across the threshold. The correction never flips the sign — a correction larger than the difference means no evidence, not evidence the other way.
p = 1 − Φ(z) for “variant higher”, p = Φ(z) for “variant lower”The tail is taken on the side chosen before the data, never on whichever side the result happened to fall. Picking the direction afterwards halves every p-value and is the most common way a one-sided A/B test gets misused, so naming the wrong direction here returns a p near 1 rather than quietly reporting the other tail.
A checkout test ran for two weeks. The control saw 12,500 users and 500 of them converted; the variant saw 12,480 users and 575 converted. You want the two-sided answer at the conventional 5% level.
4.00% against 4.61%: an absolute difference of 0.61 percentage points, a relative effect of 15.2%, z = 2.33 and p = 0.0198. The 95% interval on the difference runs from 0.10 to 1.12 percentage points, and the interval on the relative effect from 2.3% to 29.4%.
Significant, and much less precise than the headline suggests. The point estimate is a 15% relative improvement; the data is consistent with anything from 2% to 29%. If the business case needs 10% to justify the engineering cost, this test has not established it — the interval contains values well below that. Plan against the low end rather than the point estimate: a result that reached significance is likelier to have been helped by noise than hurt by it, which is most of why measured uplift differs from what shows up after rollout.
Start with the confidence interval on the difference, not the p-value. Both settle the yes/no question — the interval excludes zero exactly when p is below α — but only the interval says which effects the data is consistent with. An observed +0.6 points with an interval of [+0.05, +1.15] is significant and almost uninformative; +0.1 points with an interval of [−0.15, +0.35] is not significant and genuinely useful, because it rules out anything you would have acted on. That is the core recommendation in reporting A/B test results properly.
The p-value is the probability of a difference at least this large if both variants truly converted at the same rate. It is not the probability that the variant is better, not the probability the result replicates, and 0.049 and 0.051 are the same evidence.
Be careful with the relative effect, which is what stakeholders remember. Its interval is asymmetric and wider than the point estimate suggests, and the estimate is biased upward among tests that reached significance — a winner is likelier to have been helped by noise than hurt by it.
A non-significant result is not a finding of no effect. It is a finding that this much data could not separate the two, and whether that is useful depends entirely on the interval: if both ends are too small to act on, you have learned something real. If one end is still commercially large, the test was underpowered and the honest write-up says so rather than "no impact" — we design experiments where that is settled before launch rather than discovered after.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
We read experiments where the metric, the decision rule and the threshold were agreed before anyone looked at the data. A/B Testing