What is alpha spending?

An experiment has a fixed budget of false-positive risk, conventionally 5%. A single test at the end spends all of it in one look. Alpha spending divides that budget across a planned schedule of interim analyses — perhaps four looks at 25%, 50%, 75% and 100% of the target sample — so each check uses a stricter threshold than 0.05 and the total across all of them still comes to 5%. Stopping early becomes legitimate because the arithmetic was arranged for it in advance.

The shape of the spending function is a genuine design choice. An O'Brien-Fleming boundary spends very little early and most at the end: the first look might require p below 0.0001, the last close to 0.045. That means early stopping happens only for dramatic effects, and the final analysis is barely penalised relative to a fixed test. A Pocock boundary spends evenly, using the same threshold at every look — around 0.016 for four looks — which makes stopping early much easier and costs noticeably more power at the end.

For most product experiments O'Brien-Fleming is the better default, and the reason is asymmetric. Stopping early is a bonus rather than the objective; what you cannot afford is a large power penalty on the final analysis, which is where the majority of experiments will actually be decided. Pocock earns its place when early stopping is genuinely valuable — when continuing is costly or ethically difficult, which is why it appears more often in clinical trials than in product work.

The schedule has to be set before the experiment starts, and this is where implementations most often go wrong. Adding an unplanned look, or shifting the timing of a planned one after seeing interim data, breaks the arithmetic the boundaries were computed from. Modern spending-function formulations are more forgiving about the exact timing than the original group-sequential tables, but the number of looks and the spending shape must still be committed in advance.

Compared with always-valid inference, this is the more efficient option for a small number of planned checkpoints and the more rigid one. Alpha spending gives you four or five looks with less sample premium; always-valid inference gives unlimited unplanned looks for 20 to 40% more data. The choice follows from how the organisation actually operates rather than from which is theoretically superior.

A team plans a test needing 200,000 users per arm and wants the option to stop early. They compare four-look O'Brien-Fleming and Pocock boundaries against a fixed-horizon design, simulating both a true 6% lift and no effect at all.

The last row is the decision. Pocock's higher early-stopping rate looks attractive until you notice what it costs on the experiments that do not stop — 74.6% power against 80%, which means one experiment in twenty that would have found a real effect now misses it. Since most experiments run to the horizon, that penalty applies to the majority while the benefit applies to the minority. O'Brien-Fleming inverts that: it stops early only when the effect is large enough to be obvious, and preserves essentially full power for everything else. For product experimentation, where continuing a test costs calendar time rather than anything serious, that is the right trade. Pocock makes sense when continuing is genuinely expensive or harmful — a clinical trial where patients are receiving an inferior treatment — which is a situation product teams rarely face.