A secondary metric is one measured alongside the primary to explain what happened, without carrying authority over the decision. Its job is to describe the mechanism behind a result, not to supply an alternative verdict when the primary comes back flat.
A well-designed experiment measures far more than it judges. The primary metric decides the outcome; secondary metrics exist to answer the next question, which is why. If conversion rose, was it because more people reached the checkout or because more of those who reached it completed? Those are different mechanisms with different implications for what to build next, and only the secondaries can distinguish them.
The discipline is that they explain and do not decide. This sounds obvious and is violated constantly, in a specific and recognisable way: the primary comes back flat, someone scans the other numbers, one of them is significant, and a story forms around it. Nobody set out to cheat. But an experiment powered for one metric and then judged on whichever of fifteen looked best is not a 5% test — it is closer to a 50% one, and the reported p-value describes a procedure that was not followed.
The right mental model is that secondaries are hypotheses rather than findings. A significant movement in one is genuinely interesting and genuinely not evidence, because it was selected by being interesting. If it matters, the honest response is to run a new experiment with that metric nominated as primary and powered accordingly. That test either confirms it or does not, and either way the answer means something. Skipping that step is how organisations accumulate a body of confidently held beliefs that no experiment ever actually established.
There is a real cost to formalising them, which is why teams resist it. Applying a multiple comparisons correction across fifteen secondaries makes almost nothing significant, and that feels like throwing information away. The usual resolution is a tiered structure: the primary is tested at the full 5%, guardrail metrics are checked for harm at their own thresholds, and secondaries are reported with effect sizes and confidence intervals but no significance claim at all. That last choice is deliberate — an interval communicates the size and uncertainty of what was observed without asserting a discovery.
The practical failure mode to watch for is the metric that contradicts the primary. Conversion up, revenue per user down, and the two are describing a change that attracted less committed buyers. That is not a secondary being noisy; it is the mechanism showing itself, and it usually means the primary was too narrow rather than that the secondary should be ignored. Deciding in advance which secondaries would change your mind if they moved is the cheapest guard against reading them selectively.
The arithmetic that matters here is about how quickly a false positive becomes likely once several metrics are read, and what a correction costs.
1 − ( 1 − α )^mm metrics at α = 0.05: five gives 23%, fifteen gives 54%. The number that makes the case for not judging on secondaries.
α_adjusted = α / mFifteen secondaries at 5% overall means testing each at 0.0033 — strict enough that almost nothing survives, which is the honest cost.
largest k where p₍ₖ₎ ≤ ( k / m ) · αControls the false discovery rate rather than the family-wise rate. More power, and the right choice for exploratory panels — see multiple comparisons.
effect ± CI, no significance claimCommunicates magnitude and uncertainty without asserting a discovery the design cannot support.
A checkout test is powered for conversion rate, its primary, at 90,000 users per arm. Conversion comes back at +0.9% relative, p = 0.31 — nothing. The analyst then reviews the fourteen secondary metrics the platform reports by default, looking for an explanation.
Two secondaries cleared the uncorrected threshold. Neither survives a correction, and finding two among fourteen is close to exactly what chance predicts.
The temptation here is to write "the test lifted add-to-cart by a significant margin" and ship. The arithmetic says otherwise: with fourteen metrics there was a 51% chance of at least one false positive before any data was collected, and 0.7 was the expected count. Getting two is unremarkable. That does not make the add-to-cart movement worthless — it makes it a hypothesis. If the team believes it, the response is a new test powered on add-to-cart as primary, which would need roughly 40,000 users per arm and settle the question properly. What would be indefensible is reporting this experiment as a win. Worth noting too that the correct conclusion about the primary is not "no effect" but "no effect detected": p = 0.31 on a +0.9% observed lift with this sample means the interval still includes effects worth having.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.