What is the false discovery rate?

The family-wise error rate asks a demanding question: what is the chance of making even one mistake across this whole set of tests? Controlling it at 5% across twenty metrics requires thresholds so strict that almost nothing survives. The false discovery rate asks something more forgiving and often more useful: of the results I end up calling significant, what proportion are wrong?

The difference in what they promise is the difference in power. FWER control says you probably made no errors at all. FDR control at 5% says that among the findings you flagged, about one in twenty is a false alarm — you do not know which — and in exchange it flags many more of the real effects. For a screening exercise where the output is a prioritised list, that is a much better trade than a guarantee of purity on a nearly empty list.

The standard method is Benjamini-Hochberg, a step-up procedure. Sort the p-values ascending, compare the kth against (k/m) × α, find the largest k where the comparison passes, and declare that one and everything smaller significant. The thresholds start as strict as Bonferroni's for the smallest p-value and relax as you move up the list, which is where the extra power comes from.

It also adapts to how much is genuinely there, which is a property worth understanding. If none of the nulls are false, BH behaves essentially like Bonferroni and rejects almost nothing. If many real effects are present, the procedure notices — a lot of small p-values push the boundary up — and becomes correspondingly generous. That self-calibration is what makes it well suited to exploratory panels, where you do not know in advance how many effects to expect.

The right use in experimentation is on secondary and diagnostic metrics rather than on the primary. A primary metric decides a shipping decision and warrants strict control; a panel of fifteen supporting metrics is a screening problem, where FDR gives a defensible way to say which movements are worth investigating without either correcting them into oblivion or reporting them raw. Note that BH assumes independence or a common form of positive dependence — usually reasonable for experiment metrics, which tend to move together — and the Benjamini-Yekutieli variant handles arbitrary dependence at a cost in power.

An experiment reports twenty secondary metrics for diagnosis. Six of them have small p-values: 0.001, 0.004, 0.011, 0.019, 0.032 and 0.041. The rest are scattered above 0.1. The team compares Bonferroni and Benjamini-Hochberg at a 5% level.

The gap is smaller here than the reputation of the two methods suggests, and that is the self-calibration working. BH becomes generous when there are many small p-values pushing the boundary up; here only two are genuinely small, so it stays close to Bonferroni and rejects the rest. Had the six p-values been 0.001 through 0.012 rather than trailing up to 0.041, BH would have flagged all six while Bonferroni still flagged one. The procedure is reading how much signal appears to be present and adjusting. What matters for interpretation is that the two flagged metrics are a prioritised shortlist rather than established findings — the FDR guarantee says about one in twenty of what you flag is wrong, not that these two are individually confirmed. Either becomes a real finding by being the pre-registered primary metric of a follow-up test.