This calculator helps you calculate confidence intervals for your lift and p-values, giving you both an estimate of the effect size and an assessment of whether the observed effect is statistically significant or could plausibly be due to chance.
Choose your KPI type
A yes/no outcome for each user (e.g., converted or not).
| p-value | — |
|---|---|
| Lift | — |
| Confidence interval (lift) | [—, —] |
Use this calculator when you want to analyze a binary outcome — for example, converted or not, signed up or not, or active on day 7 — and you are comparing two independent groups.
Check for a sample ratio mismatch before you read any further. If the split is significantly different from what you configured, something upstream is broken — and you cannot proceed to this analysis until it's resolved, because no amount of statistics downstream repairs a comparison between groups that were never actually randomised.
The calculation assumes:
The calculator computes the observed conversion rate in each group, then reports two related but independently calculated quantities: the lift, with its confidence interval via the log risk-ratio method, and the absolute difference, with its own confidence interval via the standard normal approximation for two proportions. Let μ_c and μ_t denote the control and test conversion rates, and n_control and n_test the respective sample sizes.
Var_c and Var_t measure how much each group's conversion rate could vary by chance. By default, we assume both groups convert at the same underlying rate and use one shared (pooled) estimate; tick the box to let each group use its own instead.
z measures how many standard errors the observed lift is from zero (no difference). The p-value converts that into a probability: how likely a difference this large would be if there were really no effect. For a two-sided test, a small p-value and an interval that excludes zero always agree — they are the same calculation. For a one-sided test, that is no longer guaranteed: the p-value can clear your significance level while the interval — always two-sided, see below — still includes zero.
The math is done on a log scale and converted back at the end. That keeps the interval from ever going below −100% (you cannot lose more conversions than you had) and naturally makes it wider on the upside than the downside, which is how lift actually behaves. The reported lift itself is exp(log(RR)) − 1, numerically the same as (μ_t − μ_c) / μ_c. This interval is always two-sided, even for a one-sided test — a one-sided version would have no upper or lower limit at all, which is not useful to show.
This interval uses each group's own variance, not a shared pooled estimate, and is calculated separately from the lift interval above. So the two numbers will not match up exactly — that is expected, not an error.
A p-value below your chosen significance level (α) indicates the observed difference is unlikely to have occurred by chance alone — the difference between the two groups is statistically significant. If the difference is in the desired direction, this supports moving forward with implementation. If the p-value is at or above α, there is insufficient support for a true difference between the two groups, and implementation should not proceed based on this result alone.
The confidence intervals give a range of plausible values for the true lift and the true absolute difference — the narrower the interval, the more precise your estimate. Statistical significance does not necessarily imply practical significance; consider the size of the estimated lift alongside the p-value when deciding whether to implement a change.
Suppose your control group had 4,000 conversions out of 100,000 users (4.0% conversion rate), and your test group had 4,200 conversions out of 100,000 users (4.2% conversion rate).
Conversion rates of 4.00% and 4.20% — an absolute difference of +0.20 percentage points and a relative lift of +5.0%. The test yields a p-value of approximately 0.024, which is below α = 0.05. The 95% confidence interval for the lift is approximately [0.64%, 9.55%], and for the absolute difference approximately [0.03, 0.37] percentage points.
Since the p-value is below α, the result is statistically significant: there is sufficient evidence of a difference between the groups that is unlikely to be due to chance alone. Since the observed effect favors the test version, you could consider moving forward with implementation. Note how much less precise the estimate is than the headline: the point estimate is a 5% improvement and the data is consistent with anything from 0.6% to 9.6%.
Use this calculator when you want to analyze a continuous outcome — for example, revenue per user, session duration, or time on page — and you are comparing two independent groups.
Check for a sample ratio mismatch before you read any further. If the split is significantly different from what you configured, something upstream is broken — and you cannot proceed to this analysis until it's resolved, because no amount of statistics downstream repairs a comparison between groups that were never actually randomised.
The calculation assumes:
The calculator computes the observed mean in each group, then reports two related but independently calculated quantities: the lift, with its confidence interval via the log-ratio method, and the absolute difference, with its own confidence interval via the standard t-distribution approach. Let μ_c and μ_t denote the control and test group means, s²_c and s²_t their sample variances, and n_control and n_test the respective sample sizes.
By default (Welch's), each group keeps its own variance, and the degrees of freedom come from the Welch–Satterthwaite equation — usually not a whole number, which is normal, not a rounding error. Tick the equal-variances box to pool both groups into one shared variance instead, which gives the simpler df = n_control + n_test − 2.
t measures how many standard errors the observed difference is from zero (no difference). The p-value turns that into a probability: how likely a difference this large would be if there were really no effect. For a two-sided test, a small p-value and an interval that excludes zero always agree. For a one-sided test, the p-value can clear your significance level while the interval — always two-sided, see below — still includes zero.
This interval is always two-sided, even for a one-sided test — a one-sided version would have no upper or lower limit at all. On a two-sided test, it excludes zero exactly when the p-value clears your significance level.
The math is done on a log scale and converted back at the end, using the same t-critical value and degrees of freedom as the interval above. That keeps the interval from ever going below −100% and naturally makes it wider on the upside than the downside, which is how lift actually behaves. The reported lift itself is exp(log(R)) − 1, numerically the same as (μ_t − μ_c) / μ_c. Both means need to be positive for this to work; on a metric that can go negative, read the absolute difference instead.
A p-value below your chosen significance level (α) indicates the observed difference is unlikely to have occurred by chance alone — the difference between the two groups is statistically significant. If the difference is in the desired direction, this supports moving forward with implementation. If the p-value is at or above α, there is insufficient support for a true difference between the two groups, and implementation should not proceed based on this result alone.
The confidence intervals give a range of plausible values for the true lift and the true absolute difference — the narrower the interval, the more precise your estimate. Statistical significance does not necessarily imply practical significance; consider the size of the estimated lift alongside the p-value when deciding whether to implement a change.
Suppose your control group had a mean revenue per user of $50 (SD = $120) across 15,000 users, and your test group had a mean revenue per user of $53 (SD = $125) across 15,000 users.
Means of $50.00 and $53.00 — an absolute difference of +$3.00 and a relative lift of +6.00%. The test yields a t-statistic of 2.1204 on 29,948.15 degrees of freedom, corresponding to a p-value of 0.0340, which is below α = 0.05. The 95% confidence interval for the absolute difference is [0.2269, 5.7731], and for the lift is [0.44%, 11.86%].
Since the p-value is below α, the result is statistically significant: there is sufficient evidence of a difference between the groups that is unlikely to be due to chance alone. Since the observed effect favors the test version, you could consider moving forward with implementation. Read the interval before the headline, though: the point estimate is a 6% improvement and the data is consistent with anything from 0.4% to 11.9%.
Use this calculator when you want to analyze a ratio metric — a value calculated as one quantity divided by another, measured per randomization unit — for example, revenue per user, clicks per impression, or items purchased per session, and you are comparing two independent groups.
Check for a sample ratio mismatch before you read any further. If the split is significantly different from what you configured, something upstream is broken — and you cannot proceed to this analysis until it's resolved, because no amount of statistics downstream repairs a comparison between groups that were never actually randomised.
The calculation assumes:
If you upload raw data, the calculator first computes each group's ratio and standard deviation using the delta method. Once both are available for each group — whether computed that way or entered directly — it reports two related but independently calculated quantities: the lift, with its confidence interval via the log-ratio method, and the absolute difference, with its own confidence interval on the untransformed scale. Let R_c and R_t denote the control and test group ratios, SD_c and SD_t their standard deviations, and n_control and n_test the respective sample sizes.
N and D are one randomization unit's numerator and denominator (e.g., one user's revenue and sessions), and Cov(N, D) measures how much the two move together. This matters because they are usually correlated — a user with more sessions tends to generate more revenue — and ignoring it would give the wrong variance. SD(R) is not the standard deviation of either column on its own; that is why the upload path exists, to compute it correctly.
By default, each group keeps its own variance. Tick the equal-variance box to pool both groups into one shared variance instead. This uses a z rather than a t, because the delta method is already an approximation — adding a finite-sample correction on top of it would be false precision, not more accuracy.
z measures how many standard errors the observed difference is from zero (no difference); the p-value turns that into a probability. This interval is always two-sided, even for a one-sided test — a one-sided version would have no upper or lower limit at all. On a two-sided test, the interval excludes zero exactly when the p-value clears your significance level; on a one-sided test, that is no longer guaranteed.
The math is done on a log scale and converted back at the end, using the same z-critical value and variance estimates as the interval above. That keeps the interval from ever going below −100% and naturally makes it wider on the upside than the downside, which is how lift actually behaves. The reported lift itself is exp(log(R)) − 1, numerically the same as (R_t − R_c) / R_c. Both ratios need to be positive for this to work.
A p-value below your chosen significance level (α) indicates the observed difference is unlikely to have occurred by chance alone — the difference between the two groups is statistically significant. If the difference is in the desired direction, this supports moving forward with implementation. If the p-value is at or above α, there is insufficient support for a true difference between the two groups, and implementation should not proceed based on this result alone.
The confidence intervals give a range of plausible values for the true lift and the true absolute difference — the narrower the interval, the more precise your estimate. Statistical significance does not necessarily imply practical significance; consider the size of the estimated lift alongside the p-value when deciding whether to implement a change.
Suppose in your control group, users generated an average of $100 in revenue (numerator) across an average of 4.0 sessions (denominator), with a numerator variance of 40,000, a denominator variance of 3.0, and a covariance between them of 250, across 15,000 users. In your test group, users generated an average of $108 in revenue across an average of 4.075 sessions, with a numerator variance of 42,000, a denominator variance of 3.1, and a covariance of 260, across 15,000 users.
The delta method turns those moments into a control ratio of $25.00 with a standard deviation of 42.848, and a test ratio of $26.503 with a standard deviation of 42.784 — the two numbers above that nothing but the delta method produces. The test then yields an absolute difference of +$1.503 and a lift of +6.01%, with a z-statistic of 3.0402 and a p-value of 0.0024, which is below α = 0.05. The 95% confidence interval for the absolute difference is [0.5341, 2.4721], and for the lift is [2.09%, 10.08%].
Since the p-value is below α, the result is statistically significant: there is sufficient evidence of a difference between the groups that is unlikely to be due to chance alone. Since the observed effect favors the test version, you could consider moving forward with implementation.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
We read experiments where the metric, the decision rule and the threshold were agreed before anyone looked at the data. A/B Testing