TL’DR: Bayesian A/B testing falls short
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
What is a credible interval?
A credible interval is a range that contains the true effect with a stated probability, given the model and the data. It says the thing most people wrongly believe a confidence interval says, and it says it only as well as the prior behind it.
In plain English
A 95% credible interval is a range with a 95% probability of containing the true effect, given the prior and the data. That is a direct statement about the quantity you care about, and it is what almost everybody believes they are asserting when they report a confidence interval. The frequentist version cannot make that claim: it says the procedure captures the truth 95% of the time across hypothetical repeats, which is a property of the method rather than of the interval in front of you.
The distinction is real and its practical importance is easy to overstate in both directions. With a flat prior and a reasonable sample, the two intervals are frequently numerically identical to the decimal places anyone reports, so a team switching frameworks to fix their interpretation often changes nothing but the wording. Where they diverge is where the prior carries weight — small samples, rare events, or a genuinely informative prior — and there the credible interval is shifted towards the prior by an amount that depends entirely on a modelling choice.
There are two ways to construct one and they differ for skewed posteriors. An equal-tailed interval cuts 2.5% off each end, which is simple and can exclude the most probable value when the posterior is asymmetric. A highest density interval takes the shortest range containing 95% of the mass, so every point inside is more probable than every point outside — which is usually the more meaningful choice and is what most software means by HDI.
The property that makes it genuinely useful in practice is that it composes with decisions. Because the posterior is a probability distribution, you can integrate it against a threshold and ask what the chance is of clearing the effect the business case needs. A confidence interval does not support that operation without additional assumptions, and the awkward workarounds people use to approximate it are usually the credible interval's answer arrived at less directly.
The caution is that its honesty depends on the prior being stated. An interval reported without saying what prior produced it is not more transparent than a confidence interval — it is less, because the reader cannot tell how much of the range came from the data. Report the prior alongside, and where it is doing real work, report a sensitivity check across alternatives.
The formula
Two constructions, and the comparison that shows when the choice of framework changes the number.
- Equal-tailed interval
[ q₀.₀₂₅ , q₀.₉₇₅ ] of the posteriorSimple, and for a skewed posterior it can exclude the mode — the single most probable value.
- Highest density interval
shortest [a, b] with ∫ₐᵇ p(θ|data) dθ = 0.95Every point inside is more probable than every point outside. Usually the better choice for asymmetric posteriors.
- What it claims
P( a ≤ θ ≤ b | data, prior ) = 0.95A probability about the effect. The frequentist version makes a claim about the procedure instead.
- When it matches the frequentist interval
flat prior + large sample → numerically near-identicalThe divergence appears with informative priors or small samples — see the proportion calculator.
Worked example
The same conversion experiment is analysed three ways: a frequentist confidence interval, a credible interval under a flat prior, and a credible interval under an empirical prior built from 200 past experiments. Control converts 240 of 5,000; the variant 288 of 5,000.
The flat-prior credible interval essentially reproduces the confidence interval. The empirical prior halves the upper end and pulls the lower end below zero.
The first comparison is the reassuring one: with a flat prior the two frameworks agree to within a few tenths of a point, which is the usual situation and means the choice of framework rarely changes a conclusion by itself. The third row is where the substance is. An observed 20% relative lift from 5,000 users per arm is a thin result, and a prior built from 200 experiments in which effects were centred at zero with a 4% spread correctly regards 20% as extraordinary — so it pulls the estimate hard towards the mass of past experience. Which interval to believe depends on whether you accept that the past 200 experiments are informative about this one, which is a substantive question rather than a statistical one. What the table makes clear is that the empirical prior is doing most of the work here, and reporting only that interval without saying so would hide the most important input to the answer.
Common misconceptions
- “A credible interval is just a confidence interval computed differently.”
- They make different claims. A credible interval assigns 95% probability to the effect lying in the range; a confidence interval describes a procedure that captures the truth 95% of the time across repeats. They frequently coincide numerically under a flat prior, which is why the distinction is easy to dismiss and still worth knowing.
- “Credible intervals are more honest because they say what people mean.”
- They say it conditional on a prior, and an interval reported without that prior is less transparent rather than more. The claim is only as good as the modelling choice behind it, so the prior belongs in the report alongside the interval — particularly when the sample is small enough for it to matter.
- “Equal-tailed and highest-density intervals are interchangeable.”
- They coincide for a symmetric posterior and differ when it is skewed, where an equal-tailed interval can exclude the most probable value entirely. For relative lifts and other bounded or asymmetric quantities the HDI is usually the more meaningful construction, and it is worth checking which one your software returns.
Frequently asked questions
What is the practical difference from a confidence interval?
In interpretation, a great deal: the credible interval assigns probability to the effect being in the range, which is the statement people actually want. In numbers, often nothing — with a flat prior and a decent sample the two agree closely. The difference becomes material when the prior carries real weight, which means small samples, rare events, or a deliberately informative prior.
Should I use an equal-tailed interval or an HDI?
The highest density interval when the posterior is skewed, which for relative lifts it usually is, because it guarantees every value inside is more probable than every value outside. Equal-tailed is simpler and fine for symmetric posteriors. The failure mode worth avoiding is an equal-tailed interval on a skewed posterior that excludes the mode, which reads as an error to anyone who notices.
What should I report alongside a credible interval?
The prior that produced it, and a sensitivity check if it is doing much of the work. An interval quoted without its prior conceals the most important input, since a reader cannot tell how much of the range came from the data. Where the conclusion holds across a flat, empirical and sceptical prior, say so — that is a stronger statement than any single interval.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.