TL’DR: Bayesian A/B testing falls short
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
What is expected loss?
Expected loss is how much you should expect to give up by shipping a particular arm, averaged over everything the posterior still considers possible. It weights being wrong by how wrong, which makes it a decision rule rather than a measure of evidence.
In plain English
Every other quantity in this group measures evidence. Expected loss measures consequence. It asks: if I ship this arm and it turns out not to be the best, how much will I have given up — and averaged over everything the posterior still regards as possible, what is that in expectation? The answer comes out in the units of the metric, which is what makes it the closest thing here to a number a business can act on directly.
The reason it beats a probability threshold is that it weights errors by their size. A 12% chance of being wrong matters enormously if being wrong costs half a percentage point of conversion, and barely at all if being wrong costs 0.01. A rule like "ship at 95% probability to be best" treats those two situations identically, because it only counts how often you would be wrong and never how much it would cost.
It also handles multi-arm tests gracefully, which probability-based rules do not. When two variants are near-identical and both clearly beat control, P(best) splits between them and neither clears a threshold. Expected loss recognises that shipping either costs almost nothing relative to the unknown best, so both are acceptable and the choice between them can be made on other grounds — engineering cost, maintainability, whatever else matters.
Setting the tolerance is the part that requires judgement rather than statistics, and it is a feature. Deciding that you will ship when the expected loss falls below 0.05 percentage points of conversion is a business statement about what a mistake is worth, argued in units everyone in the room understands. Compare that with defending a 95% threshold, which is a convention nobody can justify from first principles. The threshold is still a choice; it is simply a choice about something people can reason about.
The caveats are the same ones that apply to everything built on a posterior. It inherits the prior, so a loss computed under an unexamined flat prior is only as trustworthy as that default. And it is not safe under an optional stopping rule — watching expected loss continuously and shipping the moment it drops below tolerance selects favourable moments exactly as peeking does in any other framework.
The formula
One integral over the region where the decision turns out to be wrong, weighted by how wrong it is.
- Expected loss from shipping B
E[ loss ] = ∫ max( θ_A − θ_B , 0 ) · p(θ_A, θ_B | data) dθOnly the region where A is actually better contributes, weighted by the margin. Computed by sampling.
- Why it beats a probability rule
P(wrong) × E[ magnitude | wrong ]A probability threshold uses only the first factor. The second is what decides whether the error matters.
- The decision rule
ship when E[ loss ] < toleranceTolerance set in metric units — a business statement rather than a statistical convention.
- In money
E[ loss ] × traffic × value per conversionConverts directly into an annual figure — see the A/B test sample size calculator for the planning side.
Worked example
Two experiments both come back at 92% probability that the variant beats control, which is below a 95% shipping threshold. They differ only in how much is at stake. Baseline conversion is 4%, annual traffic is 12 million, and a conversion is worth £28.
Identical probabilities, and the cost of the two decisions differs by a factor of fifteen — with the second test's cost of inaction ten times its cost of action.
A 95% probability threshold rejects both experiments identically, and that is clearly the wrong call on the second one. The row that settles it is the fifth: not shipping Test 2 has an expected cost of nearly £2 million a year, against £205,000 for shipping it and being wrong. When the asymmetry is that large, waiting for more certainty is itself the expensive decision. Test 1 is the opposite case — the whole effect is worth little either way, so the sensible response is to stop spending traffic on it rather than to run it longer. That comparison is what probability thresholds cannot express: they measure how sure you are and never what being sure is worth. Two cautions. Both figures inherit the prior, so an unexamined flat default should be checked before quoting a number in pounds to a board. And expected loss is not safe to watch continuously — the stopping rule has to be fixed in advance like any other.
Common misconceptions
- “Expected loss is another way of expressing the probability of being wrong.”
- It combines that probability with the magnitude of the error. A 20% chance of losing 0.01 percentage points and a 20% chance of losing a full point have the same probability and wildly different expected losses. The magnitude term is the entire reason to prefer it as a decision rule.
- “A low expected loss means the variant is better.”
- It means shipping it is unlikely to cost much, which is a different and more useful claim. Two arms that are genuinely equivalent both have near-zero expected loss, and that is the correct reading — when it does not matter which you pick, the number should say so rather than demanding a winner.
- “Because it is decision-theoretic, you can monitor expected loss continuously.”
- Shipping the first time it drops below tolerance is an optional stopping rule and selects favourable moments exactly as watching a p-value does. The framework does not change the arithmetic of repeated inspection. Fix the horizon in advance, or use a procedure explicitly built for continuous monitoring.
Frequently asked questions
How do I choose the expected loss tolerance?
Express it in metric units and convert to money: an expected loss of 0.05 percentage points on your traffic and conversion value is some annual figure, and the question is whether that is an acceptable cost of moving faster. That makes it a business decision argued in comprehensible units, rather than a defence of a 95% convention nobody can justify from first principles.
Should I compute the expected loss of not shipping too?
Yes, and it is frequently the larger of the two. Holding back a change that is probably good has a cost — the improvement you are forgoing, multiplied by how long you wait — and it is routinely ignored because inaction feels like the safe choice. Comparing the two makes the asymmetry explicit, and for a large probable effect the cost of waiting usually dominates.
How does expected loss handle more than two variants?
Cleanly, which is one of its advantages. Compute the expected loss of shipping each arm relative to the unknown best; two near-identical good variants both come out near zero, correctly indicating that the choice between them barely matters. Probability to be best splits between them instead and can leave neither clearing a threshold despite overwhelming evidence against the incumbent.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.