Just do it - Measuring Complex Customer Journeys
Measuring complex user journeys is nearly an impossible task without the proper tools. Learn how MMM & Geo Tests can help advertisers measure the impact of their cross-marketing efforts.
What is the delta method?
The delta method approximates the variance of a function of random quantities by linearising it around their means. In experimentation it is what supplies a correct standard error for a ratio metric, where both the numerator and the denominator vary.
In plain English
Some metrics are not averages of anything — they are functions of averages. Clicks per session is one sum divided by another. Revenue per order, items per basket and cost per acquisition are all the same shape. There is no per-user value to take a standard deviation of, so the ordinary standard error formula has nothing to work with, and the natural improvisations are all wrong in the same direction.
The delta method solves this generally. If a metric is a smooth function of quantities whose variances you know, you can approximate its variance by taking a first-order Taylor expansion around the means — replacing the function locally with a straight line, and propagating the variances through that line. The approximation is good whenever the sample is large enough for the means to be stable, which at experiment sample sizes they always are.
For a ratio the result has three terms, and the third is the one that gets dropped. The variance of the numerator contributes, the variance of the denominator contributes, and so does the covariance between them — which for a ratio metric is substantial, because a user who generates more sessions typically also generates more clicks. Omitting that covariance is the most common implementation error, and it does not fail loudly: it produces an interval that is wrong by a factor that depends on the correlation, with nothing in the output to indicate it.
The other half of getting this right is aggregating at the randomisation unit. If users were randomised and sessions are being counted, the variances entering the formula must be computed across users, not across sessions. Doing it at session level treats one user's twenty correlated sessions as twenty independent observations, which inflates the apparent sample size and shrinks the interval — often by a factor of two or three. The delta method applied at the wrong level is just as wrong as not using it.
The alternative is the bootstrap, resampling whole users and recomputing the ratio each time. The two agree closely and have different practical profiles: the bootstrap needs no derivation and is easy to get right but costs computation, while the delta method is a closed-form expression that runs on every experiment for free. Most mature platforms implement the delta method and use a bootstrap once to validate it, which is a good pattern to copy.
The formula
The general statement, then the ratio case that accounts for nearly all of its use in experimentation.
- The general form
Var( g(X) ) ≈ [ g'(μ) ]² · Var(X)Linearise the function at the mean and push the variance through. Accurate whenever the mean is stable, which large samples guarantee.
- The ratio case
Var(X/Y) ≈ ( 1/μ_Y² )Var(X) + ( μ_X²/μ_Y⁴ )Var(Y) − ( 2μ_X/μ_Y³ )Cov(X, Y)Three terms. The covariance is the one implementations omit, and it is usually large because numerator and denominator move together.
- The unit rule
compute every variance and covariance ACROSS RANDOMISATION UNITSUsers, not sessions. Aggregating at the wrong level understates the interval by roughly √(observations per user).
- The check
compare against a cluster bootstrap onceThey should agree closely. A large disagreement usually means a missing covariance term — see bootstrap.
Worked example
An analytics team implements a standard error for clicks per session. They have 52,000 randomised users generating 214,000 sessions and 396,000 clicks. Three implementations are compared: session-level naive, user-level without the covariance term, and the full delta method.
Three answers spanning a factor of 2.8. Only the third is correct, and a cluster bootstrap independently returns 0.0093.
The two wrong answers fail in opposite directions, which is worth understanding because it means neither is a safe approximation. The session-level version is far too small — it treats 214,000 correlated sessions as independent when only 52,000 users were randomised, so it claims roughly four times more information than the experiment collected. The user-level version without covariance is too large, because with a correlation of 0.83 the numerator and denominator largely move together and much of their individual variability cancels; ignoring that overstates the uncertainty. Only the full three-term formula gets it right, and the bootstrap agreeing to within one per cent is what confirms the implementation rather than the theory. The practical lesson is that a delta-method implementation should always be validated against a bootstrap once, because both of the failure modes shown here produce plausible-looking numbers.
Common misconceptions
- “The delta method is an exotic technique for unusual metrics.”
- It is the standard way any mature experimentation platform computes standard errors for ratio metrics, which are among the most common metrics there are — clicks per session, revenue per order, items per basket. If a platform reports intervals on those at all, it is either using this or a bootstrap.
- “The covariance term is a refinement that can be skipped.”
- It is usually the largest correction of the three. Numerator and denominator are strongly correlated in almost every real ratio metric, so omitting it substantially overstates the variance. The result is an interval that is too wide, which is safer than too narrow but still wrong, and it costs real power on every experiment.
- “Because it is an approximation, the delta method is less trustworthy than exact methods.”
- The approximation is a first-order Taylor expansion, and its error shrinks as the sample grows. At experiment sample sizes it agrees with a bootstrap to within a per cent or two. The far bigger source of error in practice is aggregating at the wrong unit, which is an implementation mistake rather than a limitation of the method.
Frequently asked questions
Which metrics need the delta method?
Any metric that divides one random total by another — clicks per session, revenue per order, cost per acquisition, items per basket. The distinguishing feature is that no single user contributes one value to the metric; the metric is computed from two sums. Ordinary averages where each user contributes one number, such as revenue per user, do not need it.
Should I implement the delta method or just bootstrap?
The delta method for production, because it is a closed-form expression that costs almost nothing per experiment, and a bootstrap once to verify the implementation. That pairing catches the two errors that matter — a missing covariance term and aggregating at the wrong unit — both of which produce plausible numbers that no amount of code review reliably spots.
What level should the variances be computed at?
The randomisation unit, always. If users were randomised, every variance and covariance in the formula is computed across users, even when the metric counts sessions or events. Computing them at event level treats correlated observations as independent and shrinks the interval by roughly the square root of the events per user, which is commonly a factor of two or more.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.