Sequential testing is a family of methods that let you analyse an experiment repeatedly while it runs without inflating the false-positive rate. They work by spending the error budget across looks, so a result can be acted on as soon as the evidence is strong enough.
A conventional experiment is a single question asked once: fix the sample size, wait, look at the result. That design is efficient and it is also unnatural, because everyone wants to know early whether something is badly broken or obviously working. The trouble is that looking repeatedly and stopping at the first significant reading turns a 5% false-positive rate into roughly 25% over a fortnight of daily checks. Each look is another chance to cross the threshold, and the threshold was priced for one.
Sequential methods fix this by making the threshold a function of when you look. Instead of one fixed critical value, you get a boundary that starts very strict early — when little data has accumulated and noise is largest — and relaxes towards the conventional value at the planned end. The total probability of crossing it anywhere across all looks is exactly the significance level you chose. The error budget is spent deliberately rather than accidentally.
There are two broad families. Group sequential designs, which come from clinical trials, specify a fixed number of interim analyses in advance and use an alpha-spending function — O'Brien-Fleming is the common one, and it is very conservative early, which suits a setting where stopping means declaring a drug effective. Always-valid or anytime-valid methods, built on mixture sequential probability ratio tests and confidence sequences, allow unlimited looks at any moment, which fits a dashboard that anybody can refresh.
None of this is free. A sequential design that permits continuous monitoring needs a larger maximum sample than a fixed-horizon test — commonly 10-25% more — because the early strictness has to be paid for somewhere. The trade is that the *expected* duration is often shorter, since large effects are detected and stopped early. If your effects are usually marginal, you will pay the premium and rarely collect; if they are often decisive or occasionally catastrophic, you collect regularly.
The clearest case for it is not speed but safety. A guardrail metric that catches a serious regression needs to be watched continuously, and waiting two weeks to discover that a release broke checkout is not an option. Running guardrails on a sequential boundary while the primary metric runs to a fixed horizon is a reasonable and common arrangement — and far better than the usual compromise, which is watching everything daily and pretending the arithmetic still holds. We work through the practical version in navigating sequential testing.
The unifying idea is that the critical value depends on how much information has accumulated. These are the three formulations you will meet.
P(any look significant) ≈ 1 − (1 − α)^k for weakly dependent looksLooks on accumulating data are strongly correlated, so the true figure is lower than this bound — but simulation puts fourteen daily looks at α = 0.05 near 0.25, five times nominal.
α*(t), α*(0) = 0, α*(1) = αt is the information fraction, from 0 to 1. The function decides how much of the budget each interim look may spend.
α*(t) = 2 − 2·Φ( z₁₋α/₂ / √t )Spends almost nothing early and nearly all of it at the end, so the final analysis is close to a conventional test. The default for most group sequential designs.
P( ∀t: θ ∈ CIₜ ) ≥ 1 − αCoverage holds simultaneously at every moment, not just at a pre-planned one, which is what makes unlimited peeking legitimate. Wider than a fixed-horizon interval at the same n.
A team plans a 14-day test with a fixed sample of 120,000 per arm. They want to monitor it, so they compare three approaches: check daily and use the ordinary 0.05 threshold; use an O'Brien-Fleming boundary with five planned interim looks; or run the fixed test and look once at the end.
Naive peeking runs at five times the stated error rate. The sequential design holds it at 5% and costs 10% more maximum sample, while stopping early whenever the effect is large.
Look at the boundary at the first interim: a p-value of 0.000005 is required to stop after a fifth of the data. That is not a technicality, it is the point — a difference that looks impressive on day three routinely evaporates, and the boundary encodes exactly how much scepticism day three deserves. By the final look the threshold has relaxed to 0.041, almost the conventional value, so a sequential design costs very little at the end. The 10% sample premium is the price of the option to stop early, and whether it is worth paying depends on your effects: a portfolio where occasional changes are strongly positive or actively harmful collects on that option regularly, while one of uniformly marginal effects pays the premium and almost never exercises it.
Peeking and early stop is one of the most common mistakes companies do across the board. Learn how sequential testing in the Frequentist approach allows it while maintaining control over false positive rates.
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.