What kind of metric is this?

Before choosing a test, classify the metric. A binary outcome happened or it did not: converted, churned, clicked. A count records how many times: sessions per user, articles read, support tickets raised. A continuous quantity can take any value in a range: revenue, time on page, load latency. The three behave differently enough that treating them alike is the source of a good deal of quietly wrong analysis.

The most useful thing the classification buys you is knowing where the variance comes from. For a binary metric, variance is fully determined by the rate — p(1 − p) and nothing else — which is why no separate estimate is needed and why a z-test applies. It also means the noise is not something you can reduce by measuring more carefully; at a 3% conversion rate the coefficient of variation is about 5.7, and that is a property of the metric rather than of your instrumentation. Continuous metrics are the opposite: their variance has to be estimated from the data, which is what the t-distribution exists to account for.

Counts sit awkwardly between the two and are the most frequently mishandled. They are non-negative, usually skewed, and their variance tends to grow with their mean rather than staying fixed. The classical model is Poisson, where variance equals the mean — but real product counts are almost always overdispersed relative to that, because users differ enormously in how much they do anything. Sessions per user is not Poisson; it is a mixture of very light and very heavy users, and applying Poisson assumptions understates the variance and produces confidence intervals that are too narrow.

The sample-size consequences are large and worth internalising. A binary metric at a low base rate is far noisier than its tidy appearance suggests, and detecting a 5% relative lift on a 3% conversion rate takes roughly 100,000 users per arm. A continuous metric with a long right tail — revenue per user, where a handful of customers dominate — can need more still, which is why revenue is so often relegated to a secondary metric despite being what the business runs on.

One practical move follows from all this. Converting a skewed continuous metric into a binary one — "spent more than £50" rather than "amount spent" — throws away magnitude information and frequently produces a more sensitive test, because it removes the tail that was driving the variance. Whether that trade is worth making depends on whether the business question is about how many customers cross a threshold or about total revenue, and that is a decision to make before the test rather than after seeing which version was significant.

A team can measure the same checkout change three ways: as a binary conversion (did the user purchase), as a count (number of items purchased), or as a continuous amount (revenue per user). They want to detect a 5% relative improvement at 80% power and need to know what each choice costs in traffic.

Revenue is the metric the business actually cares about and the one it can least afford to test on. At this traffic the conversion version reads in about three weeks and the revenue version in three months, by which point the answer is no longer useful. That gap is why conversion is so often the primary and revenue a guardrail, and it is a compromise rather than a preference. Two things are worth checking before accepting it. First, whether the revenue tail can be tamed: capping at a sensible percentile typically cuts the required sample substantially and changes the question only slightly. Second, whether conversion and revenue actually move together for this kind of change — if the variant converts more people at lower basket values, optimising conversion is optimising against the business, and only the slower metric would ever reveal it.