TL’DR: Bayesian A/B testing falls short
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
What is probability to be best?
Probability to be best is the posterior chance that a given arm has the highest true value among those tested. It is the headline number most Bayesian testing tools report, and it says nothing about how much better that arm is.
In plain English
Probability to be best answers a natural-sounding question: given everything observed, what is the chance this arm really is the winner? It is computed by sampling from each arm's posterior many times and counting how often each comes out on top. Most Bayesian testing tools show it as the headline figure, because it reads as plain English in a way a p-value never does.
The immediate limitation is that it carries no magnitude. An arm can be 97% likely to be best by a margin of 0.02 percentage points, and the number looks identical to an arm that is 97% likely to be best by five points. For a shipping decision the margin is usually what matters — a change that is almost certainly better by an amount too small to fund is not a change worth making — and P(best) is structurally silent on it.
The second limitation appears with more than two arms and surprises people. Probabilities must sum to one across the arms, so two near-identical good variants split the credit between them. Three arms where B and C are both clearly better than A but indistinguishable from each other might read 5% / 48% / 47%, and neither B nor C reaches a conventional threshold despite the evidence being strong that A should lose. The number is answering "which single arm is best" when the decision is usually "should we move off the incumbent".
Both problems are solved by asking the decision-relevant question instead. The probability of clearing a threshold — P(effect > 2%) — restores the magnitude, and expected loss restores the cost. Expected loss also handles the multi-arm case gracefully, because shipping B when C is marginally better carries almost no cost, which is exactly what the near-tie means and what P(best) obscures.
The practical guidance is to treat P(best) as a communication device rather than a decision rule. It is genuinely good at conveying that a result is or is not clear-cut to an audience that misreads p-values. It is a poor basis for a threshold, and a rule of the form "ship at 95% probability to be best" inherits both limitations and adds the peeking problem if the number is watched continuously.
The formula
One integral, computed by sampling, plus the two quantities that fix its blind spots.
- The quantity
P( θ_k = max over all arms )Computed by drawing from every arm's posterior repeatedly and counting wins.
- The constraint
Σ P(best) over arms = 1Why two equally good arms split the credit, and neither reaches a high threshold.
- The magnitude-aware version
P( θ_B − θ_A > Δ )Substitute the effect the business case needs. Restores what P(best) drops.
- The decision-aware version
E[ loss from choosing arm k ]Weights being wrong by how wrong — see expected loss and the ANOVA calculator for the frequentist multi-arm case.
Worked example
A four-arm test on a 5% conversion baseline. Control performs clearly worst; variants B and C are close together and clearly better; variant D sits in between. The team reads P(best) and then recomputes the decision-relevant quantities.
No arm exceeds 45% probability to be best, and the evidence that control should lose is overwhelming.
A rule of "ship at 95% probability to be best" would ship nothing here, which would be plainly wrong — two variants are each about 98% likely to beat control, and the reason neither reaches a high P(best) is that they are competing with each other rather than with the evidence. That is the multi-arm splitting problem in its ordinary form. The last row resolves the decision cleanly: shipping B costs an expected 0.06 percentage points against the unknown true best, which is negligible, so the choice between B and C barely matters and both are clearly preferable to control. The generalisable reading is that P(best) answers "which one" when the decision is usually "should we move", and once there are more than two arms those two questions come apart. Report P(best) if the audience finds it intuitive, and decide on expected loss.
Common misconceptions
- “A 95% probability to be best means the variant is 95% likely to improve things.”
- It means it is 95% likely to be the best of those tested, which is a comparison among arms rather than a claim about magnitude. The improvement could be negligible. The quantity that answers the intended question is the probability of clearing a threshold you care about.
- “If no arm reaches a high probability to be best, the test is inconclusive.”
- Not necessarily — two similar strong arms split the credit, so neither reaches a high figure while the evidence against control is overwhelming. Check the probability each arm beats control individually, which is not subject to the splitting, before concluding nothing was learned.
- “Probability to be best is a Bayesian replacement for the p-value.”
- It is a different quantity with its own blind spots rather than a strict improvement. It ignores magnitude, behaves awkwardly with several arms, and is not safe under an optional stopping rule. Its real advantage is that non-specialists read it correctly, which is a communication benefit rather than a statistical one.
Frequently asked questions
What threshold should I ship at?
Preferably not one based on this number. A rule like "ship at 95%" fails in multi-arm tests where good variants split the credit, and it ignores the size of the improvement entirely. Set the threshold on expected loss instead — ship when the expected cost of being wrong falls below what you would tolerate — which handles both problems and expresses the decision in units someone can argue about.
Why does no variant reach a high probability with several arms?
Because the probabilities sum to one across arms, so two similarly good variants divide the credit between them. Three arms could read 5 / 48 / 47 while the evidence that the first should lose is overwhelming. Look at each arm's probability of beating control individually, which is not subject to the split, and use expected loss to choose between the leaders.
How does it relate to a p-value?
For two arms under a flat prior, the probability of beating control corresponds closely to one minus the one-sided p-value — a 97% probability lines up with p = 0.03. They are near-equivalent restatements of the same evidence. The differences appear with informative priors, with more than two arms, and in how readily each is misread by a non-specialist audience.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.