Sweet Test O' Mine: Rocking the truth in experimentation
Our "Sweet Test o' Mine" webinar debunked some of the most common A/B testing myths, highlighting the importance of testing often, iterating quickly, and trusting data over intuition.
What is an A/A test?
An A/A test splits traffic between two identical experiences and checks that the platform reports no difference. It validates the measurement machinery rather than any product change, and a significant result means something in the pipeline is broken.
In plain English
An A/A test runs the experiment machinery with nothing to measure. Users are assigned to two arms, both receive exactly the same experience, and the results are analysed as though it were a real test. Since there is no difference to find, anything the platform reports is telling you about the platform rather than about the product.
It catches a specific and valuable class of problem: bugs in the measurement rather than in the product. Assignment that is not evenly random, metric definitions that compute differently across arms, event tracking that drops data on one variant path, users who end up in both arms through a cookie or identity bug, and standard errors that are simply too small for the metric's correlation structure. Each of those produces confidently wrong results on every subsequent experiment, and none is visible from a normal test where a real effect could be doing the work.
The most useful version is not a single A/A test but a distribution of them. Run several hundred by repeatedly splitting historical data and collect the p-values: if the platform is sound, they should be roughly uniform between 0 and 1, and about 5% should fall below 0.05. That last point is the one that trips teams up — an A/A test coming back significant is not automatically a failure, because at the 5% level one in twenty will by definition. What indicates a problem is *too many* significant results, or a p-value distribution that clusters rather than spreading evenly.
The mistake worth avoiding is running one A/A test, seeing p = 0.03, and launching an investigation into a healthy platform. The mistake in the other direction is running one, seeing p = 0.6, and concluding everything is fine — a single non-significant result tells you very little. The distribution is the diagnostic; a single run is close to uninformative.
For the specific question of whether assignment is balanced, a sample ratio mismatch check is far more sensitive and can run continuously on every live experiment rather than as an occasional exercise. A/A testing is the broader instrument, covering metric computation and variance estimation as well as assignment, and it is best used when standing up a new platform, after a significant change to the pipeline, or periodically as a scheduled health check.
The formula
Nothing new is computed. What matters is what the distribution of results should look like when everything is working.
- The expectation
p-values ~ Uniform( 0, 1 )Under a true null the p-value is uniform by construction. Any departure from flat is a finding about the pipeline.
- How many should be significant
P( p < 0.05 ) = 0.05One in twenty A/A tests SHOULD reach significance. A run of them all coming back clean is itself suspicious.
- The variance check
observed SD of effects ≈ reported standard errorIf observed spread exceeds the reported SE, the platform is understating uncertainty — common with ratio metrics.
- The assignment check
chi-square goodness of fit against the intended splitMore sensitive than an A/A test for balance alone, and runnable continuously — see the chi-square calculator.
Worked example
A team stands up a new experimentation platform and validates it by running 500 simulated A/A tests over historical data, on two metrics: conversion rate, an ordinary per-user binary, and clicks per session, a ratio metric.
Conversion behaves exactly as it should. The ratio metric produces false positives at nearly four times the nominal rate.
The two metrics diagnose different things and only one is broken. Conversion's 5.4% significant is precisely what a healthy platform gives, and the flat distribution confirms it — this is what passing looks like, and note that it includes 27 significant results. The ratio metric is a clear failure, and the last row identifies the cause: the reported standard error is less than half the spread actually observed, which is the signature of computing variance at session level when randomisation happened at user level. Every experiment run on this metric would have been overstating its confidence by a factor of two. Had the team validated on conversion alone, they would have shipped a platform that was correct for ordinary metrics and systematically wrong for every ratio metric — which is the more insidious outcome, because those are the metrics teams reach for when a simple average will not do.
Common misconceptions
- “A significant A/A test means the platform is broken.”
- At the 5% level, one in twenty A/A tests will be significant by definition — that is what the threshold means. A single significant run is expected noise. What indicates a problem is an excess of them across many runs, or a p-value distribution that is not roughly uniform.
- “One clean A/A test validates the platform.”
- A single non-significant result is close to uninformative, since a broken platform still produces plenty of them. The diagnostic is the distribution across many runs. Several hundred simulated A/A tests on historical data cost almost nothing and are what actually establish that the machinery works.
- “A/A tests are the way to check that assignment is balanced.”
- A sample ratio mismatch check is far more sensitive for that specific question and can run automatically on every live experiment. A/A testing covers a wider surface — metric computation, variance estimation, tracking — and is better suited to validating a new pipeline than to routine monitoring.
Frequently asked questions
How many A/A tests do I need to run?
Several hundred, which is only feasible by simulating them over historical data rather than running them live. Repeatedly split a past period at random, compute the result as though it were an experiment, and collect the p-values. Five hundred runs gives a clear enough picture of the distribution to spot a platform inflating its false-positive rate, and costs compute time rather than traffic.
What kinds of problems does an A/A test actually find?
Uneven assignment, metrics computed differently across arms, tracking that drops events on one variant path, users appearing in both arms through identity bugs, and — most commonly in practice — standard errors that are too small because variance was computed at the wrong unit. That last one is invisible in a normal experiment, where a real effect masks the overconfidence.
When should I run A/A tests?
When standing up a new platform, after any significant change to the assignment or metrics pipeline, when adding a new metric type such as a ratio metric, and as a scheduled periodic check. Running one before every experiment is unnecessary and consumes traffic that should be measuring something. The simulated version is cheap enough to run on a schedule.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.