What is survivorship bias?

Analyse the customers you still have and you learn about the customers who stayed. If leaving was related to the thing you are measuring — and it usually is — then the surviving group is systematically different from the group you started with, and every statistic computed from it inherits that difference. This is survivorship bias, and its distinguishing feature is that the data looks complete.

It is a specific mechanism within the broader problem of selection bias, and what makes it worth its own treatment is that the selection happens after the fact and over time. Ordinary selection bias is about who entered; survivorship is about who remained. That temporal quality is what makes it so easy to miss, because the sample was perfectly fine when it was assembled.

In product analytics the everyday form is a satisfaction or engagement metric that improves while the business deteriorates. Users who found the product frustrating leave, so the average satisfaction among remaining users rises — the metric records an improvement caused entirely by the departure of the unhappy. Any measure computed over an active user base has this property, which is why a rising average engagement figure during a period of elevated churn should be treated as a warning rather than a result.

In experiments it appears when attrition differs between arms. If the treatment causes more of the disengaged users to leave, the treatment arm's remaining users are a healthier group than control's, and comparing them measures who survived rather than what the treatment did. The diagnostic is straightforward: compare attrition rates between arms as a matter of routine, alongside the sample ratio mismatch check, since differential dropout is the same class of failure.

The remedies depend on the setting. For cohort analysis, fix the cohort at entry and follow it, so departures are visible as departures rather than silently excluded — a metric over a fixed cohort can fall, which is the point. For time-to-event questions, the log-rank test and survival methods handle censoring properly. For experiments, analyse everyone assigned rather than everyone remaining, which is intent-to-treat and is the same discipline applied to a different exit.

A subscription product reports average weekly sessions per active user, which rises steadily over six months and is presented as evidence that engagement work is succeeding. An analyst recomputes it over a fixed cohort — everyone active in month one, followed regardless of whether they remain.

The mechanism is in the last row: among users who stayed, engagement did rise slightly — the headline is not a computational error. What it omits is that a growing share of the original cohort stopped being counted at all, and those were disproportionately the light users. Rising churn mechanically improves any average taken over survivors, so the metric was reporting the deterioration as an improvement. The fixed-cohort figure is the honest one for the question being asked, and it falls because leavers contribute zero rather than disappearing. Two things follow. Any per-active-user metric should be read alongside the churn rate, since the two move together for reasons that have nothing to do with product quality. And a metric that cannot go down when the business gets worse is not measuring the business.