Running experiments
The vocabulary of A/B testing: what you fix before a test starts, what you watch while it runs, and the diagnostics that say the result cannot be trusted.
Most of what decides whether an experiment gives you a usable answer is settled before it starts. What gets randomised, at what level the results are analysed, how traffic is split, and which metric the decision hangs on are all choices made in advance, and each of them can quietly invalidate everything downstream. The terms in this group are those choices and the checks that catch them when they go wrong.
The distinction that causes the most damage in practice is between the unit you randomise and the unit you analyse. Randomise by user and analyse by session and the arithmetic underneath every significance test stops holding, because sessions from one user are not independent of each other. The result is a p-value that looks fine and is not, which is the worst kind of wrong number — see randomization.
The second is knowing when a running test is already broken. A sample ratio mismatch is the single highest-value diagnostic in experimentation: a split that came back 49.2/50.8 when it should have been even means something upstream is dropping users non-randomly, and nothing computed from that data is worth reading. It is also cheap to check and almost never checked.
Where a term has a calculator, the entry links to it, so you can go from a definition to a number without leaving the site. We build experimentation programmes for a living — see our A/B testing work.
Terms in this group
A/B Testing at Bell Statistics
We design and run experimentation programmes, from the assignment layer up to the readouts leadership actually acts on. See how we work.