The log-rank test compares two time-to-event curves in full rather than at a single cut-off — the difference between "do these users churn more slowly?" and "how many had churned by day 30?". Plan the study in events, the currency this test actually spends, then get the chi-square statistic, the p-value and the hazard ratio.
| Events (Schoenfeld) | 631 |
|---|---|
| Events (Freedman) | 636 |
| Schoenfeld, before rounding | 630.52 |
| Patients, arm 1 | 1,148 |
| Patients, arm 2 | 1,148 |
| Patients, total | 2,296 |
| Event probability per patient | 27.50% |
The requirement is 636 events, not 2,296 patients. The patient count is only what it takes to produce those events at the 27.5% event rate you entered: halve that rate and the events needed do not move while the enrolment roughly doubles. Schoenfeld's formula asks for 631 events and Freedman's for 636; they are different approximations to the same distribution and they separate as the hazard ratio moves away from 1, so plan on the larger — 636. The levers that matter, in order, are the length of follow-up and the definition of the event — not the recruitment target.
| Hazard ratio to detect | Required events |
|---|---|
| 0.40 | 38 |
| 0.46 | 52 |
| 0.51 | 71 |
| 0.57 | 99 |
| 0.63 | 143 |
| 0.68 | 215 |
| 0.74 | 342 |
| 0.79 | 596 |
| 0.85 | 1,212 |
| 0.91 | 3,348 |
| p-value | 0.0258 |
|---|---|
| Chi-squared | 4.9679 |
| Degrees of freedom | 1 |
| Hazard ratio (O/E) | 1.3776 |
| 95% confidence interval | [1.041, 1.823] |
| Total observed events | 196.000 |
| Total expected events | 196.000 |
The two survival curves differ somewhere over the follow-up you observed (p 0.0258, on 196 events). Somewhere is all the test says — not when, and not whether the gap persists; curves that separate early and rejoin later describe a delay rather than a durable improvement, and only the Kaplan–Meier plot distinguishes them. The observed-over-expected hazard ratio is 1.38, with the data consistent with anything from 1.04 to 1.82. Read that as a ratio of rates: at any moment, an arm-1 patient who has survived that long faces that multiple of the arm-2 patient's risk. It is not a difference in how many people eventually have the event, which over a fixed window is usually a much smaller number.
| Hazard ratio (arm 1 relative to arm 2) | Estimate | Interval |
|---|---|---|
| Hazard ratio | 1.38 | 1.04 to 1.82 |
Use the log-rank test when the outcome is how long something takes, not whether it happened by an arbitrary date. Time to churn, time to first purchase, how long a trial survives before converting or lapsing — all share a feature that breaks ordinary tests: when you analyse, most users have not had the event yet. Someone who signed up three days ago is not a non-churner but a churner-so-far, and this test handles that censoring natively rather than scoring it as a zero.
The usual alternative is to pick a horizon and compare two proportions — day-30 retention, say — with a chi-square test or the A/B test calculator. That is a fine test of a very specific question, and it throws away when the events happened. If one arm loses users in week one and the other the same number over eight weeks, the day-30 proportions are identical while the experience is not.
It tells you whether two curves differ, not how much or where. For a magnitude, fit a Cox proportional-hazards model — the log-rank test is its score test — or read the hazard ratio this page estimates from observed and expected counts. Three or more arms extend to a k−1 degree-of-freedom version, as one-way ANOVA extends a two-group comparison of means. If the question is how time-to-event relates to a continuous covariate, that is regression rather than a correlation test.
Two assumptions carry real weight here, and one of them decides whether the test can find anything at all:
At every distinct time an event occurs, the test compares how many events fell in group 1 with how many would be expected if the group label carried no information, given who is still at risk. Summing observed minus expected across those times gives one statistic, chi-square on one degree of freedom. The consequence that surprises people is that the sample-size tab asks for events, not subjects: those who never have the event contribute to the risk sets and almost nothing to the power. Doubling a cohort of users who never churn buys nothing.
e₁ⱼ = dⱼ · n₁ⱼ / nⱼdⱼ events occur at time j among nⱼ subjects at risk, of whom n₁ⱼ are in group 1. Under the null, events split in proportion to the risk sets.
χ² = (Σⱼ (d₁ⱼ − e₁ⱼ))² / Σⱼ vⱼ, vⱼ = dⱼ(nⱼ−dⱼ)n₁ⱼn₂ⱼ / (nⱼ²(nⱼ−1))One degree of freedom. vⱼ is the hypergeometric variance, which is what makes tied event times correct rather than merely tolerable.
HR ≈ (O₁/E₁) / (O₂/E₂)The standard rough estimate. A Cox model gives a better one with a proper interval, but this comes free with the test.
e = (z₁₋α/₂ + z₁₋β)² / (p₁p₂(ln HR)²)p₁ and p₂ are allocation proportions, so p₁p₂ = 0.25 for an even split and this reduces to 4(z₁₋α/₂ + z₁₋β)²/(ln HR)². The default here, and what most survival power software implements.
e = (z₁₋α/₂ + z₁₋β)² (1 + r·HR)² / (r(1 − HR)²)Shown alongside because the two disagree. At HR = 0.80 they are within about 1%; at HR = 0.40 Freedman asks for roughly 14% more events. Quoting whichever is smaller without saying which is how survival studies end up underpowered.
n = e / P(event during follow-up)The step everyone skips. If only 30% of users will have churned by the end of the window, you need more than three times as many subjects as events, and lengthening the window is often cheaper than recruiting more.
A subscription product is testing a new onboarding sequence and expects it to slow churn rather than eliminate it. The hypothesis is a hazard ratio of 0.80 — a 20% lower instantaneous churn risk at any point in the first 90 days. About 30% of new subscribers historically cancel within 90 days, and the split is even. You want 80% power at the two-sided 5% level.
631 churn events under Schoenfeld, 636 under Freedman — the two essentially agree at this hazard ratio. At a 90-day churn rate near 30% that is roughly 2,100 subscribers in total, about 1,050 per arm, each followed for the full 90 days.
The 631 is the real requirement and the 2,100 is a consequence of it. Change nothing but the churn rate — run the same test on an enterprise tier where 90-day churn is 6% — and the events needed are unchanged while the subscribers needed rise to around 10,500. Power lives in the events, and a population that does not churn cannot supply them however many you enrol. The levers worth pulling are follow-up and event definition: extending the window to 180 days might take the event probability to 45% and cut the cohort by a third, at a cost in calendar time rather than traffic.
A small p-value says the two curves differ somewhere over the follow-up you observed. It does not say where, by how much, or that the difference persists. Always publish the two Kaplan–Meier curves next to it: curves that separate at week one and rejoin at week six describe a delay, not a retention improvement, and only the picture tells you which you bought.
The hazard ratio is the effect size, and it is a ratio of rates rather than a difference in outcomes. An HR of 0.80 means that at any moment a treated user is 20% less likely to churn than an otherwise-similar control who has survived as long. It does not mean 20% fewer people churn: over a fixed window it usually becomes a much smaller difference in the eventual proportion, and skipping that conversion reliably over-promises — the gap we describe in why the uplift in A/B tests often differs from real-world results.
A large p-value needs most care. Check the curves for crossing first: power against differences that reverse partway through is very poor, and a null on crossing curves is an artefact of the statistic rather than a finding about users. Check the event count too — 90 events against a design that needed 631 means the study is not yet readable.
Resist looking every week and stopping when it goes significant. Survival data accumulates continuously, which makes repeated inspection unusually tempting and unusually damaging. Use a group-sequential boundary with the looks planned in advance, as in navigating sequential testing, or fix the event count and read it once. Specifying a survival design before it runs is squarely what our causal inference work covers.
Peeking and early stop is one of the most common mistakes companies do across the board. Learn how sequential testing in the Frequentist approach allows it while maintaining control over false positive rates.
Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.
A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.
Retention questions are survival questions. We design them in events, follow-up and censoring rather than in a flat user count. Causal Inference Analysis