Bell Statistics

Running experiments

The vocabulary of A/B testing: what you fix before a test starts, what you watch while it runs, and the diagnostics that say the result cannot be trusted.

20 terms

Most of what decides whether an experiment gives you a usable answer is settled before it starts. What gets randomised, at what level the results are analysed, how traffic is split, and which metric the decision hangs on are all choices made in advance, and each of them can quietly invalidate everything downstream. The terms in this group are those choices and the checks that catch them when they go wrong.

The distinction that causes the most damage in practice is between the unit you randomise and the unit you analyse. Randomise by user and analyse by session and the arithmetic underneath every significance test stops holding, because sessions from one user are not independent of each other. The result is a p-value that looks fine and is not, which is the worst kind of wrong number — see randomization.

The second is knowing when a running test is already broken. A sample ratio mismatch is the single highest-value diagnostic in experimentation: a split that came back 49.2/50.8 when it should have been even means something upstream is dropping users non-randomly, and nothing computed from that data is worth reading. It is also cheap to check and almost never checked.

Where a term has a calculator, the entry links to it, so you can go from a definition to a number without leaving the site. We build experimentation programmes for a living — see our A/B testing work.

Terms in this group

  • A/A test

    Testing nothing against nothing to check the instrument — and it should be significant about five per cent of the time.

  • A/B testing

    A randomised experiment on live traffic — and randomisation is the only part that makes it causal.

  • A/B/n test

    Several complete alternatives against one control — and each extra arm costs twice: less traffic and another chance to be wrong.

  • Analysis unit

    The mismatch that makes confidence intervals two or three times too tight, with nothing in the output to show it.

  • Bucketing

    Three properties assignment must have — random, deterministic, independent — and what breaks when each one fails.

  • Cluster randomization

    Assign the group, not the person — the remedy for interference, paid for in statistical power.

  • Concurrent testing

    Overlapping tests are safe for each result and blind to the combination — which is where the surprise lands.

  • Control group

    The counterfactual, made concrete — and the reason a before-and-after comparison is not an experiment.

  • Experimentation platform

    Assignment, exposure, analysis and the guardrails — and the last of those is what separates a platform from a flag service.

  • Exposure point

    Assigned is not the same as exposed — and the gap between them is where most of your effect size goes.

  • Feature flag

    The switch experiments run on — and the reason a flag that changes mid-test quietly invalidates the result.

  • Global holdout

    The only measurement that can tell you whether a year of wins added up — and it usually says they did not.

  • Hash-based assignment

    Compute the variant instead of storing it — stateless, consistent everywhere, and free of a lookup on every request.

  • Holdout group

    A control that outlives the experiment — the only way to see what a change did after month one.

  • Multivariate test

    Every combination, so interactions are visible — at roughly four times the traffic per effect you want to see.

  • Randomization

    The one mechanism that makes a comparison causal — and the four ways it silently fails.

  • Randomization unit

    What gets assigned decides what counts as independent — and that decides whether your p-values mean anything.

  • Switchback test

    Randomise the clock instead of the users — the answer when everyone shares one supply pool.

  • Traffic allocation

    50/50 is not caution, it is the optimum — and a 90/10 split needs nearly three times the traffic.

  • Treatment group

    The users who get the change — and the counting rule that decides whether the comparison is still randomised.

A/B Testing at Bell Statistics

We design and run experimentation programmes, from the assignment layer up to the readouts leadership actually acts on. See how we work.

Other topics

Back to the glossary