Just do it - Measuring Complex Customer Journeys
Measuring complex user journeys is nearly an impossible task without the proper tools. Learn how MMM & Geo Tests can help advertisers measure the impact of their cross-marketing efforts.
What is a ratio metric?
A ratio metric divides one total by another — clicks per session, revenue per order — where both parts vary. Because the denominator is itself random and correlated with the numerator, the usual standard error formula understates the true uncertainty.
In plain English
Clicks per session. Revenue per order. Items per basket. These look like ordinary averages and are not, and the difference has bitten most experimentation platforms at some point. An ordinary average has a fixed denominator — the number of users you randomised — and each user contributes one value. A ratio metric divides one sum by another sum, where both sums vary between arms and are correlated with each other.
The problem this creates is specific. The standard error formula everyone reaches for assumes the denominator is a known constant, and treats the ratio as though it were a mean over sessions. But sessions are not the randomisation unit — users are — and a variant that changes how many sessions people have has changed the denominator as well as the numerator. The resulting interval is too narrow, sometimes by a factor of two or three, and nothing in the output indicates that anything is wrong.
The mechanism worth understanding is the mismatch between analysis unit and randomisation unit. If you randomise users and analyse sessions, the sessions belonging to one user are not independent of each other — a heavy user contributes twenty correlated observations. Treating them as twenty independent data points inflates your apparent sample size enormously, and the confidence interval shrinks accordingly. The number of independent observations is the number of users, not the number of sessions, and the analysis has to respect that.
There are two standard fixes and they agree closely in practice. The delta method derives an approximate variance for the ratio analytically, accounting for the variance of both parts and the covariance between them; it is fast and is what most mature platforms implement. The bootstrap resamples whole users and recomputes the ratio each time, which handles the correlation structure without any derivation at all. The bootstrap is easier to get right and slower; the delta method is the production choice.
There is also an alternative that avoids the problem entirely: change the metric to one whose denominator is the randomisation unit. Clicks per user rather than clicks per session, revenue per user rather than revenue per order. That makes each user contribute exactly one number, restores an ordinary average, and lets the standard machinery work. The cost is that the metric now blends two effects — whether people do more sessions and whether each session is better — which may or may not be what you wanted to measure.
The formula
The naive formula and the correct one, and the covariance term that separates them. That term is what a per-session analysis of a per-user experiment silently omits.
- The metric
R = Σ Xᵢ / Σ YᵢA ratio of two sums over users, not the average of per-user ratios. Those two quantities differ and are frequently confused.
- The naive standard error
SE ≈ s_X / ( ȳ · √n_sessions )Treats the denominator as fixed and sessions as independent. Both are false, and both push the interval the same way — too narrow.
- Delta method variance
Var(R) ≈ ( 1/ȳ² )·Var(X) + ( x̄²/ȳ⁴ )·Var(Y) − ( 2x̄/ȳ³ )·Cov(X, Y)The covariance term is the one that matters: numerator and denominator move together, and ignoring it distorts the estimate.
- The bootstrap alternative
resample USERS with replacement, recompute R each timeNo derivation needed and the correlation is handled automatically — see bootstrap.
Worked example
A content site tests a related-articles module on clicks per session. The experiment randomises 88,000 users, who between them generate 412,000 sessions. Control averages 1.84 clicks per session, the variant 1.93 — a 4.9% lift. The platform reports the result using a per-session standard error.
Both analyses find a positive effect. The naive interval is 2.7 times narrower than the correct one and its p-value is more than an order of magnitude smaller.
The conclusion survives here, which is what makes this pattern dangerous — the naive analysis is wrong and still reaches the right verdict, so nothing prompts anyone to check it. The damage shows up in what gets built on the number. A team reading +3.1% to +6.7% will forecast confidently from roughly 5%; the honest range starts at 0.4%, which is a materially different business case. And in the very common situation where the effect is smaller, the naive interval excludes zero while the correct one does not, and a false positive ships. The root cause is visible in the second row: 88,000 users is the amount of independent information, not 412,000 sessions, and the naive method is claiming 4.7 times more data than the experiment collected.
Common misconceptions
- “Clicks per session is just an average, so a t-test on sessions is fine.”
- Sessions from one user are correlated, so treating them as independent observations inflates the apparent sample size by the average sessions per user. The interval comes out too narrow by roughly the square root of that factor. The independent unit is whatever you randomised, and the analysis has to be built on it.
- “The average of each user's ratio is the same as the ratio of the totals.”
- They are different quantities and can point in different directions. The ratio of sums weights users by how many sessions they had, so heavy users dominate; the average of per-user ratios gives every user equal weight. Neither is wrong, but they answer different questions and the choice should be deliberate rather than an accident of the SQL.
- “If the delta method is complicated, the naive interval is close enough.”
- It is not close — factors of two to three are ordinary — and it always errs towards overconfidence rather than caution. If implementing the delta method is awkward, the bootstrap over users needs no derivation at all and gets the same answer. What is not acceptable is knowingly reporting an interval that is several times too tight.
Frequently asked questions
Should I use the delta method or a bootstrap for ratio metrics?
The delta method for production, because it is an analytic formula and costs almost nothing to compute on every experiment. The bootstrap for verification or for metrics too awkward to differentiate, since it needs no derivation and handles the correlation structure automatically. They agree closely at experiment sample sizes, so running the bootstrap once to validate a delta method implementation is a cheap and worthwhile check.
Can I avoid ratio metrics altogether?
Often, by redefining the metric so the denominator is the randomisation unit — clicks per user instead of clicks per session. Each user then contributes one number and the ordinary machinery applies. The trade-off is that the redefined metric blends two effects: whether users have more sessions and whether each session is better. If you specifically need the per-session quality, the ratio is the right metric and needs the right variance.
Does variance reduction work on ratio metrics?
Yes, and it is one of the places CUPED pays off most, because ratio metrics are noisy to begin with. The implementation needs care: the adjustment has to be applied in a way that respects the ratio structure rather than to the numerator alone, which is a known trap. Most platforms that support both have solved it, but it is worth confirming rather than assuming, since getting it wrong biases the estimate rather than merely weakening it.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.