What is a proxy metric?

The outcomes a business runs on are slow. Lifetime value takes a year to observe, retention takes months, and churn announces itself only after the customer has gone. Experiments run in two or three weeks. A proxy metric bridges that gap: something measurable now — activation, first purchase, week-one engagement — used as a stand-in for the thing that matters, on the understanding that improving it improves the real outcome later.

Nearly every experimentation programme depends on one, and the dependency is usually unexamined. The assumption being made is precise and strong: that an intervention which moves the proxy will move the outcome, in the same direction, by a predictable amount. That is a causal claim about how the two are connected, and correlation between them in historical data is not sufficient evidence for it. Users who activate in week one do retain better — but that is largely because engaged people do both, not because activation causes retention.

The distinction matters because it decides what happens when you intervene. If activation and retention share a common cause — user intent — then a change that raises measured activation without changing intent moves the proxy and nothing else. This is the surrogate paradox, and its practical form is depressingly familiar: an aggressive onboarding prompt lifts activation by 8%, the team ships it, and ninety days later retention is flat or slightly worse. Nobody lied and the metric moved exactly as promised.

What makes a proxy trustworthy is not correlation strength but validation by intervention. The check is whether past experiments that moved the proxy also moved the outcome — which requires holding a slow-outcome readout on a subset of shipped changes and looking back. Programmes that do this typically discover their proxy is directionally right and quantitatively unreliable: a 10% proxy lift buys somewhere between 2% and 6% of the real thing, with wide variance. That is still useful, and it is much weaker than the one-to-one relationship most roadmaps implicitly assume.

Two habits keep a proxy honest. Re-estimate the relationship periodically, because it drifts as the product and the user base change, and a proxy validated three years ago on a different cohort is a historical artefact. And pair it with a long-horizon check — a holdout or a slower confirmation read — so the programme has some way of noticing when the proxy has decoupled. Without that, a proxy failure is silent by construction: every experiment keeps reporting wins.

A media subscription business uses week-one article reads as a proxy for 6-month retention. The correlation across users is 0.58, which the team treats as strong validation. Over two years they ship eleven changes that significantly lifted the proxy, and a 5% long-term holdout lets them check what actually happened to retention.

This is a proxy working and being badly overinterpreted at the same time. It is directionally right — the aggregate retention effect is positive and the programme is not optimising noise. But the roadmap had been forecasting from a 1:1 relationship, so two years of expected gains were roughly four times what arrived. The four changes with no transfer are the more instructive part: reviewing them, three were prompts that manufactured reads from users who were going to churn regardless, which is the surrogate paradox in its ordinary form. Note what made all of this visible. Without the 5% holdout none of it would have surfaced, because every one of the eleven experiments reported a clean win on the metric it was judged on. The correlation of 0.58 was true throughout and told the team nothing about what interventions would do.