An A/B/n test compares a control against several variants at once, each a complete alternative rather than a combination of settings. Every extra arm splits the traffic further and adds a comparison, so both precision and the error rate degrade with n.
An A/B/n test is an A/B test with more variants: one control and several complete alternatives, each of which could ship on its own. Three headline treatments, four pricing layouts, five onboarding flows. It differs from a multivariate test, where the arms are combinations of independent settings and the point is to learn how those settings interact — here each arm is simply a different candidate.
The appeal is obvious: rather than sequentially testing four ideas over four months, test them together in one. The cost is that each additional arm charges twice. Traffic is divided further, so with five arms each holds a fifth of the users and every comparison is less precise than the same test with two. And each comparison is another opportunity for a false positive — four variants against one control is four tests, and at the 5% level that is roughly an 18.5% chance of at least one spurious winner when none of them does anything.
The correction to use depends on the question, and the default choice is usually wrong. If the question is "does any challenger beat control" — which it almost always is in product work — then the comparisons are challenger-against-baseline, and Dunnett's correction is designed for exactly that. It is meaningfully more powerful than Bonferroni, because it accounts for the fact that all comparisons share the same control arm and are therefore correlated. Tukey's is for when every pair matters, which is rarer than it is applied.
The traffic arithmetic is worth doing before committing. Five arms rather than two means each arm holds 20% rather than 50% of traffic, and the correction raises the bar each comparison must clear. Together those typically mean an A/B/n with four challengers needs somewhere between two and three times the total traffic of a simple A/B test to detect the same effect. That is often still better than four sequential tests, which take four times as long and cannot compare the variants against each other at all.
The practical failure to avoid is treating the winner's measured effect as unbiased. Selecting the best of five arms and reporting its estimate overstates it, because part of what made it the best was luck — the winner's curse. The measured lift of the winning arm is an optimistic estimate of what shipping it will deliver, and the more arms you ran, the more optimistic it is.
Two costs, both arithmetic. The traffic split is straightforward; the multiplicity correction is where the choice of method matters.
1 − ( 1 − α )^k, k comparisonsFour challengers at 5% gives 18.5%. Six gives 26.5%. The reason a correction is not optional.
n_arm = N / ( k + 1 )Five arms means each holds a fifth. Precision in every comparison falls accordingly.
compare each challenger to control at an adjusted critical valueAccounts for the shared control arm, so it is more powerful than Bonferroni for this design — see the ANOVA calculator.
E[ observed effect | selected as best ] > true effectThe winning arm's measured lift is biased upward, and more so the more arms were run.
A team has four candidate onboarding flows plus the current one. They can run a five-arm test on 300,000 users, or four sequential A/B tests of 150,000 each. Baseline activation is 22%, and they want to detect a 5% relative improvement.
The five-arm test detects a 5.1% effect in three weeks. Four sequential tests detect 3.8% and take three months.
The trade is time against sensitivity and it usually favours the multi-arm design. Twelve weeks of sequential testing buys a slightly finer detection threshold and costs a quarter, during which the market and the product both move, so the four tests are not even measuring quite the same thing. The Dunnett-versus-Bonferroni row is worth noticing: the same design and the same data, and choosing the correction designed for challenger-versus-control comparisons rather than the generic one recovers about 0.3 percentage points of sensitivity for free. The caution is on the other side of the result. If flow C wins at +6.1%, that figure is an overestimate — it was selected from four for being the largest, and some of its margin is luck. Plan on shipping something closer to the lower end of its confidence interval, or re-measure it against control alone before forecasting from it.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
When business constraints limit A/B testing treatment exposure, analysts must choose between the statistical integrity of a balanced design (e.g., 10/10) and the speed of an unbalanced design (e.g., 10/90). The unbalanced 10/90 split, which uses full traffic to dramatically accelerate the experiment, comes with risks like inflated false positives from skewed metrics, Sample Ratio Mismatch, and resource contention bias. Read on as we examine this critical trade-off to help you balance speed with statistical validity.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.