What is metric sensitivity?

A metric is sensitive when a real change of a size you care about produces a detectable movement in the time you can afford to run. That is a property of the metric rather than of the experiment, and it varies enormously: conversion rate at 20% is sensitive, revenue per user is not, and quarterly retention is so insensitive that no single experiment can move it detectably at all. Knowing where a candidate metric sits on that spectrum before designing a test is what separates a programme that learns things from one that mostly reports "no significant difference".

The quantity that predicts it is the coefficient of variation — the standard deviation divided by the mean. Required sample scales with its square, so a metric with twice the relative noise needs four times the traffic for the same relative effect. This is why the intuition that "we have millions of users so we can measure anything" fails so often. Revenue per user with a long right tail can have a coefficient of variation above ten, and at that level millions of users buys the ability to detect effects of five per cent, not one.

There are four levers and they are worth knowing in order of usual payoff. Reduce the variance directly — CUPED removes the part of each user's outcome that was predictable from their history, typically 20 to 50 per cent where history exists. Cap the tail, since a handful of extreme values can dominate the variance of a revenue metric and winsorization removes their leverage. Trigger the analysis at exposure, so users who never encountered the change are excluded and stop diluting the effect. And choose a denser metric — clicks rather than purchases, activation rather than retention — accepting that it becomes a proxy metric with everything that implies.

The triggering lever is the one most often left unused and it is frequently the largest. If a change affects only the 12% of users who reach a particular page, an experiment analysed over all assigned users is measuring an effect diluted by a factor of eight. Restricting to the exposed population multiplies the observable effect accordingly. The condition is that exposure must be defined identically in both arms — filtering on something the treatment itself influences is how a sensitivity improvement becomes a bias.

The reason to work on this at all is that sensitivity determines what a programme can learn. A team whose primary metric can only resolve effects above 8% will find almost nothing, conclude that most ideas do not work, and be unable to distinguish that from an inability to see. Improving sensitivity converts an experimentation programme from a filter that rejects everything into one that measures.

A team must detect a 3% relative improvement in a checkout change with 180,000 users per arm available over three weeks. Their primary metric is revenue per user. They evaluate whether that is feasible and what each available lever would buy.

The progression is the point. Each individual lever looks modest — capping the tail, a correlation of 0.58, restricting to the 41% who reach checkout — and together they turn an infeasible experiment into a comfortable one. Triggering does the most work here and is the one teams most often skip, because it requires exposure instrumentation that does not exist by default. Two cautions. Winsorizing at the 99th percentile has to be decided now, not after seeing results, and it changes the question slightly: the metric is no longer total revenue but revenue excluding the extreme tail, which is a defensible thing to optimise and should be stated. And triggering is only valid if checkout entry is defined identically in both arms — if the variant changes who reaches checkout, the exposed populations differ and the comparison is no longer randomised.