What is the log-rank test?

Some questions are not about whether something happened but how quickly. Did the new onboarding get people to their first successful action sooner? Do accounts on the annual plan churn later than monthly ones? Reducing those to a binary — converted within 30 days, yes or no — throws away most of what you measured and makes the answer depend entirely on where you drew the line. The log-rank test compares the whole timeline instead.

Its central problem is censoring, and censoring is why an ordinary z-test is not merely less efficient here but actively misleading. At any analysis date, some users have converted and others have not converted *yet* — and the second group is not the same as never converting. A user who signed up three days ago and has not converted carries very different information from one who signed up eight months ago and has not. Dropping them biases the result, and counting them as failures biases it the other way. The log-rank test uses each subject for exactly as long as they were observed and then stops.

The mechanism is a sequence of small comparisons. At each moment an event occurs, the test looks at everyone still at risk, works out how many events would be expected in each group if the groups were identical — proportional to how many are still at risk in each — and records the gap between expected and observed. Those gaps are accumulated across every event time and the total is compared against a chi-square distribution. It is closer in spirit to a running tally than to a single snapshot comparison.

What it assumes is proportional hazards: that one group's instantaneous risk of the event stays a roughly constant multiple of the other's over the whole period. That assumption fails in a specific and common way — when the survival curves cross. A change that speeds up early conversions but produces more churn later can look like nothing at all, because the early advantage and the late disadvantage cancel in the accumulated statistic. Plotting the two Kaplan-Meier curves before running the test is not optional; it is how you find out whether the test is answering your question.

It also tells you only that the curves differ, not by how much. For a magnitude you want a hazard ratio from a Cox proportional hazards model, or something concrete like median time to event in each group. "Churn is significantly later on the annual plan" is not a number anyone can plan around; "median tenure is 14.2 months against 9.6" is.

A subscription business tests a new activation flow on time-to-first-purchase, tracking 3,000 signups per arm over 90 days. At the analysis date 1,340 control users and 1,505 variant users have purchased; the rest are censored, most of them recent signups who simply have not had long enough.

The headline conversion difference — 44.7% against 50.2% — understates what happened, because both figures are cut off at an arbitrary analysis date with two thousand users still inside their window. The timing is where the effect really shows: nine days off the median, which for a subscription business compounds into earlier revenue and a shorter payback period. Two things to check before acting. Confirm the Kaplan-Meier curves do not cross — an activation flow that pulls purchases forward without increasing the eventual total would show a large early gap that closes, and the log-rank test would still report significance while the business case evaporated. And confirm censoring is unrelated to treatment: if the variant also increased account deletions, those users leave the risk set for a reason connected to the treatment, which is informative censoring and biases everything here.