What is a primary metric (OEC)?

Every experiment needs one number that decides it. Not a dashboard, not a balanced view of twelve indicators — one metric, nominated before launch, that the ship-or-not decision follows from. Ronny Kohavi's term for it is the overall evaluation criterion, and the two names describe the same object: the metric that has been given authority over the outcome.

The reason for insisting on one is not tidiness. Watch twelve metrics at the 5% level and roughly one in two experiments will show something significant even when the change does nothing at all — that is the multiple comparisons problem, and without a nominated primary it operates invisibly. What happens in practice is subtler than deliberate cheating: the test comes back flat on conversion, someone notices session depth is up, and a narrative forms around it. Nobody lied, and the shipped decision was still made by searching until something looked good.

A usable primary metric has to satisfy three things at once, and the tension between them is the whole difficulty. It must be sensitive enough to move within the experiment's duration, which rules out most of the outcomes that matter most. It must be aligned with the business, so improving it is genuinely good. And it must be hard to game — a metric that improves when the product gets worse is not a neutral choice, because teams optimise what is measured whether or not anyone intends them to. Clicks are wonderfully sensitive and reward making things confusing enough to require more clicking.

That sensitivity constraint is what forces the compromise most teams end up making. Retention, lifetime value and revenue per customer are what the business runs on, and none can be read in fourteen days. So the primary becomes something upstream and faster — activation, first purchase, week-one engagement — which is a proxy metric and carries the assumption that moving it moves the real thing. That assumption is worth stating explicitly and validating occasionally, because when it fails an entire programme optimises confidently in the wrong direction.

Two rules make the whole structure work. Write the metric down before the test starts, with the minimum detectable effect you designed for, so the decision rule exists before the data can influence it. And pair it with a small set of guardrail metrics that must not degrade — the primary asks whether the change helped, the guardrails ask what it cost, and a change that lifts the primary while breaking a guardrail is not a win.

A subscription team must pick a primary metric for a redesigned onboarding flow. Three candidates are on the table: day-90 retention, which is what the business cares about; first-week active days, which is faster; and clicks in onboarding, which is fastest of all. Traffic allows about 40,000 users per arm over three weeks.

Clicks is the most sensitive metric by a wide margin and the worst choice available. Its correlation with day-90 retention is 0.08, which means optimising it is very nearly optimising noise — and worse, it is gameable in the exact direction that hurts, since a more confusing flow requires more clicks. Retention is the honest objective and cannot be measured in time to decide anything. Active days is the compromise: it costs sensitivity relative to clicks, so the detectable effect is 2.4% rather than 1.0%, and it buys a defensible link to the outcome. That correlation of 0.61 is the assumption the whole programme now rests on, and it should be re-estimated on live data once or twice a year rather than treated as settled. Day-90 retention still gets measured — as a slower confirmation that the proxy is not drifting, not as the experiment's decision rule.