----
----
Conversion rate
Planning a geo test? The Bell Experiments MCP server runs the Bell Statistics engine as tools your assistant can call — geo test design and measurement today, A/B testing next.
Get the MCP serverUse this calculator when you want to analyze a binary outcome — for example, converted or not, signed up or not, or active on day 7 — and you are comparing two independent groups.
Check for a sample ratio mismatch before you read any further. If the split is significantly different from what you configured, something upstream is broken — and you cannot proceed to this analysis until it's resolved, because no amount of statistics downstream repairs a comparison between groups that were never actually randomised.
The calculation assumes:
The calculator computes the observed conversion rate in each group, then reports two related but independently calculated quantities: the lift, with its confidence interval via the log risk-ratio method, and the absolute difference, with its own confidence interval via the standard normal approximation for two proportions. Let μ_c and μ_t denote the control and test conversion rates, and n_control and n_test the respective sample sizes.
Var_c and Var_t measure how much each group's conversion rate could vary by chance, and each group uses its own: Var_c = μ_c(1 − μ_c) and Var_t = μ_t(1 − μ_t). Each term then simplifies to (1 − μ)/x, where x is that group's number of conversions, so SE[log(RR)] = √( (1 − μ_c)/x_control + (1 − μ_t)/x_test ) — Katz's standard error for the log risk ratio. The test statistic and the lift interval below both use it, which is why they always agree on a two-sided test.
z measures how many standard errors the observed lift is from zero (no difference). The p-value converts that into a probability: how likely a difference this large would be if there were really no effect. For a two-sided test, a small p-value and an interval that excludes zero always agree — they are the same calculation. For a one-sided test, that is no longer guaranteed: the p-value can clear your significance level while the interval — always two-sided, see below — still includes zero.
The math is done on a log scale and converted back at the end. That keeps the interval from ever going below −100% (you cannot lose more conversions than you had) and naturally makes it wider on the upside than the downside, which is how lift actually behaves. The reported lift itself is exp(log(RR)) − 1, numerically the same as (μ_t − μ_c) / μ_c. This interval is always two-sided, even for a one-sided test — a one-sided version would have no upper or lower limit at all, which is not useful to show.
This interval uses each group's own variance, not a shared pooled estimate, and is calculated separately from the lift interval above. So the two numbers will not match up exactly — that is expected, not an error.
A p-value below your chosen significance level (α) indicates the observed difference is unlikely to have occurred by chance alone — the difference between the two groups is statistically significant. If the difference is in the desired direction, this supports moving forward with implementation. If the p-value is at or above α, there is insufficient support for a true difference between the two groups, and implementation should not proceed based on this result alone.
The confidence intervals give a range of plausible values for the true lift and the true absolute difference — the narrower the interval, the more precise your estimate. Statistical significance does not necessarily imply practical significance; consider the size of the estimated lift alongside the p-value when deciding whether to implement a change.
Suppose your control group had 4,000 conversions out of 100,000 users (4.0% conversion rate), and your test group had 4,200 conversions out of 100,000 users (4.2% conversion rate).
Conversion rates of 4.00% and 4.20% — an absolute difference of +0.20 percentage points and a relative lift of +5.0%. The test yields a p-value of approximately 0.024, which is below α = 0.05. The 95% confidence interval for the lift is approximately [0.64%, 9.55%], and for the absolute difference approximately [0.03, 0.37] percentage points.
Since the p-value is below α, the result is statistically significant: there is sufficient evidence of a difference between the groups that is unlikely to be due to chance alone. Since the observed effect favors the test version, you could consider moving forward with implementation. Note how much less precise the estimate is than the headline: the point estimate is a 5% improvement and the data is consistent with anything from 0.6% to 9.6%.
Use this calculator when you want to analyze a continuous outcome — for example, revenue per user, session duration, or time on page — and you are comparing two independent groups.
Check for a sample ratio mismatch before you read any further. If the split is significantly different from what you configured, something upstream is broken — and you cannot proceed to this analysis until it's resolved, because no amount of statistics downstream repairs a comparison between groups that were never actually randomised.
The calculation assumes:
The calculator computes the observed mean in each group, then reports two related but independently calculated quantities: the lift, with its confidence interval via the log-ratio method, and the absolute difference, with its own confidence interval via the standard t-distribution approach. Let μ_c and μ_t denote the control and test group means, s²_c and s²_t their sample variances, and n_control and n_test the respective sample sizes.
By default (Welch's), each group keeps its own variance, and the degrees of freedom come from the Welch–Satterthwaite equation — usually not a whole number, which is normal, not a rounding error. Tick the equal-variances box to pool both groups into one shared variance instead, which gives the simpler df = n_control + n_test − 2.
The test is on the log scale: t measures how many standard errors the log of the ratio of means, log(R), is from zero (no difference), with log(R) and SE[log(R)] as defined under the lift below and the degrees of freedom above. The p-value turns that into a probability: how likely a difference this large would be if there were really no effect. For a two-sided test, a small p-value and a lift interval that excludes zero always agree, because they are the same calculation. For a one-sided test, the p-value can clear your significance level while the interval — always two-sided, see below — still includes zero.
This interval is always two-sided, even for a one-sided test — a one-sided version would have no upper or lower limit at all. It is computed on the untransformed scale rather than the log scale the test uses, so on a marginal result it can include zero while the p-value sits just under your significance level; the lift interval is the one that always agrees with the test.
The math is done on a log scale and converted back at the end, using the same t-critical value and degrees of freedom as the interval above, and the same standard error the test statistic uses. With the equal-variances box ticked, the pooled variance replaces both s²_c and s²_t in SE[log(R)] too. That keeps the interval from ever going below −100% and naturally makes it wider on the upside than the downside, which is how lift actually behaves. The reported lift itself is exp(log(R)) − 1, numerically the same as (μ_t − μ_c) / μ_c. Both means need to be positive for this to work; on a metric that can go negative, read the absolute difference instead.
A p-value below your chosen significance level (α) indicates the observed difference is unlikely to have occurred by chance alone — the difference between the two groups is statistically significant. If the difference is in the desired direction, this supports moving forward with implementation. If the p-value is at or above α, there is insufficient support for a true difference between the two groups, and implementation should not proceed based on this result alone.
The confidence intervals give a range of plausible values for the true lift and the true absolute difference — the narrower the interval, the more precise your estimate. Statistical significance does not necessarily imply practical significance; consider the size of the estimated lift alongside the p-value when deciding whether to implement a change.
Suppose your control group had a mean revenue per user of $50 (SD = $120) across 15,000 users, and your test group had a mean revenue per user of $53 (SD = $125) across 15,000 users.
Means of $50.00 and $53.00 — an absolute difference of +$3.00 and a relative lift of +6.00%. The test yields a t-statistic of 2.1209 on 29,948.15 degrees of freedom, corresponding to a p-value of 0.0339, which is below α = 0.05. The 95% confidence interval for the absolute difference is [0.2269, 5.7731], and for the lift is [0.44%, 11.86%].
Since the p-value is below α, the result is statistically significant: there is sufficient evidence of a difference between the groups that is unlikely to be due to chance alone. Since the observed effect favors the test version, you could consider moving forward with implementation. Read the interval before the headline, though: the point estimate is a 6% improvement and the data is consistent with anything from 0.4% to 11.9%.
Use this calculator when you want to analyze a ratio metric — a value calculated as one quantity divided by another, measured per randomization unit — for example, revenue per session, clicks per impression, or items purchased per session, and you are comparing two independent groups.
Check for a sample ratio mismatch before you read any further. If the split is significantly different from what you configured, something upstream is broken — and you cannot proceed to this analysis until it's resolved, because no amount of statistics downstream repairs a comparison between groups that were never actually randomised.
The calculation assumes:
If you upload raw data, the calculator first computes each group's ratio and standard deviation using the delta method. Once both are available for each group — whether computed that way or entered directly — it reports two related but independently calculated quantities: the lift, with its confidence interval via the log-ratio method, and the absolute difference, with its own confidence interval on the untransformed scale. Let R_c and R_t denote the control and test group ratios, SD_c and SD_t their standard deviations, and n_control and n_test the respective sample sizes.
N and D are one randomization unit's numerator and denominator (e.g., one user's revenue and sessions), and Cov(N, D) measures how much the two move together. This matters because they are usually correlated — a user with more sessions tends to generate more revenue — and ignoring it would give the wrong variance. SD(R) is not the standard deviation of either column on its own; that is why the upload path exists, to compute it correctly.
By default, each group keeps its own variance and the degrees of freedom come from the Welch–Satterthwaite equation — usually not a whole number, which is normal, not a rounding error. Tick the equal-variance box to pool both groups into one shared variance instead, which gives the simpler df = n_control + n_test − 2. The delta method is itself a large-sample approximation, so the t reference is a convention rather than an exact correction; at the sample sizes ratio metrics are used at, it and a z agree to the third decimal place.
The test is on the log scale: t measures how many standard errors the log of the ratio of the two ratios, log(R), is from zero (no difference), with log(R) and SE[log(R)] as defined under the lift below and the degrees of freedom above; the p-value turns that into a probability. For a two-sided test, a small p-value and a lift interval that excludes zero always agree, because they are the same calculation. For a one-sided test, the p-value can clear your significance level while the interval — always two-sided, see below — still includes zero.
This interval is always two-sided, even for a one-sided test — a one-sided version would have no upper or lower limit at all. It is computed on the untransformed scale rather than the log scale the test uses, so on a marginal result it can include zero while the p-value sits just under your significance level; the lift interval is the one that always agrees with the test.
The math is done on a log scale and converted back at the end, using the same t-critical value and variance estimates as the interval above, and the same standard error the test statistic uses. That keeps the interval from ever going below −100% and naturally makes it wider on the upside than the downside, which is how lift actually behaves. The reported lift itself is exp(log(R)) − 1, numerically the same as (R_t − R_c) / R_c. Both ratios need to be positive for this to work.
A p-value below your chosen significance level (α) indicates the observed difference is unlikely to have occurred by chance alone — the difference between the two groups is statistically significant. If the difference is in the desired direction, this supports moving forward with implementation. If the p-value is at or above α, there is insufficient support for a true difference between the two groups, and implementation should not proceed based on this result alone.
The confidence intervals give a range of plausible values for the true lift and the true absolute difference — the narrower the interval, the more precise your estimate. Statistical significance does not necessarily imply practical significance; consider the size of the estimated lift alongside the p-value when deciding whether to implement a change.
Suppose in your control group, users generated an average of $100 in revenue (numerator) across an average of 4.0 sessions (denominator), with a numerator variance of 40,000, a denominator variance of 3.0, and a covariance between them of 250, across 15,000 users. In your test group, users generated an average of $108 in revenue across an average of 4.075 sessions, with a numerator variance of 42,000, a denominator variance of 3.1, and a covariance of 260, across 15,000 users.
The delta method turns those moments into a control ratio of $25.00 with a standard deviation of 42.848, and a test ratio of $26.503 with a standard deviation of 42.784 — the two numbers above that nothing but the delta method produces. The test then yields an absolute difference of +$1.503 and a lift of +6.01%, with a t-statistic of 3.0371 on 29,998 degrees of freedom and a p-value of 0.0024, which is below α = 0.05. The 95% confidence interval for the absolute difference is [0.5340, 2.4721], and for the lift is [2.09%, 10.08%].
Since the p-value is below α, the result is statistically significant: there is sufficient evidence of a difference between the groups that is unlikely to be due to chance alone. Since the observed effect favors the test version, you could consider moving forward with implementation.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.
The Bell Experiments MCP server gives your assistant the Bell Statistics engine: ask it to pick test and control markets, estimate the budget or read the lift in plain language, and get the answer with the assumptions behind it. A/B testing is next.
We read experiments where the metric, the decision rule and the threshold were agreed before anyone looked at the data. A/B Testing
