What is a guardrail metric?

Every experiment optimises something, and almost every optimisation has a way to cheat. Aggressive email increases sessions and increases unsubscribes. A more prominent upsell increases revenue per order and increases returns. Removing a confirmation step increases completion and increases support tickets. A guardrail metric is the one you watch to make sure the win you measured is not being paid for somewhere the test was not looking.

The defining property is asymmetry. A guardrail can veto a launch; it can never justify one. That sounds like a small distinction and it is the entire mechanism — the moment a guardrail is allowed to declare a win, it has become a second primary metric, the multiple comparisons problem arrives, and the decision rule turns into "ship if anything came out green". Nominating exactly one metric that can say yes, and several that can say no, is what keeps the type I error rate roughly where you set it.

That asymmetry also changes how they should be powered, in a way most teams get backwards. On the primary metric you are protecting against a false positive, so you want a strict threshold. On a guardrail you are protecting against a missed regression, so the error that matters is the false negative — and correcting a guardrail for multiple comparisons makes it less sensitive to exactly the thing it exists to catch. Most mature programmes run guardrails uncorrected, at a looser threshold, and treat the occasional false alarm as the cost of the insurance.

Good guardrails come in three flavours. Business-critical metrics that no experiment is allowed to damage regardless of what it improves — revenue, retention, checkout completion. Quality and trust metrics that degrade slowly and expensively: unsubscribes, complaint rate, app store rating, support contacts. And validity metrics, which are not about the product at all but about whether the experiment can be believed — sample ratio mismatch, latency, error rate, crash rate. That third group catches more real problems than the other two combined.

Two practical points. Guardrails are usually the metrics worth watching with sequential testing, because waiting two weeks to learn that a release broke checkout is not a real option, and continuous monitoring is precisely what sequential boundaries are built for. And the threshold on a guardrail should be an interval question rather than a significance one: what you want to know is not whether the metric moved, but whether you can rule out a drop larger than you are willing to accept — which is an equivalence or non-inferiority test, not a significance test.

A test makes the newsletter signup prompt more prominent. Signups rise 14%, comfortably significant. Two guardrails are watched: unsubscribe rate within 30 days, and the spam complaint rate. Unsubscribes move from 2.10% to 2.31%, p = 0.09. Complaints move from 0.041% to 0.052%, p = 0.21.

Read as a significance test both guardrails say ship, and that reading is wrong in a specific way: p = 0.09 on a guardrail is not reassurance, it is an unresolved question with the evidence leaning the wrong way. The interval is what the decision needs, and it cannot rule out an unsubscribe increase four times larger than the team said it would accept. Complaints matter more still, because a rising complaint rate is scored against the sending domain and damages deliverability for every message including transactional ones — a slow, expensive harm that a two-week test measures poorly. The right call is to keep the arm running specifically to tighten the guardrail interval, and to treat the 14% signup win as unbanked until it does.