Sweet Test O' Mine: Rocking the truth in experimentation
Our "Sweet Test o' Mine" webinar debunked some of the most common A/B testing myths, highlighting the importance of testing often, iterating quickly, and trusting data over intuition.
What is Twyman's law?
Twyman's law holds that any figure which looks interesting or unusual is probably wrong. In experimentation it is the working rule that a surprisingly large result should trigger an investigation of the measurement before it triggers a celebration.
In plain English
Twyman's law is a heuristic rather than a theorem: any figure that looks interesting or different is usually wrong. It comes from media audience research and it earns its place in experimentation because the base rates support it. Most changes do very little, so a result showing a 40% improvement is more likely to be a tracking bug, a broken assignment or a filtering artefact than a genuine transformation of user behaviour.
The prior doing the work here is empirical rather than cynical. Large experimentation programmes report that the majority of tested ideas produce no detectable effect and that typical wins are in the low single digits. Against that background, an enormous result is unusual twice over — unusual as an effect, and drawn from a distribution where measurement errors are considerably more common than transformative changes. Bayes does the rest.
What makes the law useful rather than merely deflating is that the checks are cheap and ordered. Start with the sample ratio mismatch: if the split is uneven, assignment is broken and nothing else matters. Then confirm the metric is computed identically in both arms, which catches events that only fire on one variant path. Then check whether the effect is concentrated in a segment or a time window, which points at a partial outage or a bot. Then confirm the analysis unit matches the randomisation unit, since a mismatch inflates both the apparent effect and its significance.
The law cuts both ways and the negative direction is neglected. A result showing a catastrophic 30% decline is equally likely to be an instrumentation failure, and teams are much quicker to investigate a surprising loss than a surprising win. That asymmetry is itself a source of bias in what gets shipped: implausible wins get shipped and implausible losses get debugged, so errors flow in one direction.
None of this says large effects never happen. They do, particularly on new surfaces, on badly broken flows, and where a change removes a genuine obstacle. The discipline is simply that the burden of proof scales with the size of the claim — and the cost of an hour spent checking is trivial against the cost of building a roadmap on a tracking bug.
The formula
The reasoning is Bayesian and the arithmetic is worth doing once, because the conclusion is stronger than intuition suggests.
- The posterior odds
P( error | big result ) / P( real | big result ) = ( P(error)/P(real) ) × likelihood ratioWith errors more common than transformative effects, the prior odds already favour error.
- The empirical prior
most tested changes produce no detectable effect; typical wins are low single digitsReported consistently across large programmes. A 40% lift sits far outside that distribution.
- The check order
SRM → metric parity → segment and time concentration → analysis unitOrdered by how often each finds something — see the chi-square calculator for the first.
- The symmetric case
apply the same scrutiny to implausible lossesInvestigating only surprising wins lets errors flow in one direction.
Worked example
A test on a signup form reports a 47% relative lift in completed registrations, significant at p < 0.0001. The team runs the standard checks before taking it to the roadmap review.
The second check found it. A duplicate event on the variant path accounted for almost the entire measured lift.
The sample ratio passing is what makes this instructive — the most common check, the one most likely to catch a broken experiment, was clean, and the defect was one step further along. That is the argument for running all four rather than stopping at the first pass. The corrected result is a 2.1% lift that is not statistically distinguishable from zero, which is an ordinary and unexciting outcome and almost certainly the truth. Worth noting what would have happened without the check: a 47% figure is large enough to reshape a roadmap, and the error would have surfaced months later as an unexplained gap between forecast and reality, at which point tracing it back to a duplicate event in one experiment would have been very difficult. An hour against that is not a close call.
Common misconceptions
- “Twyman's law is just pessimism about experimentation.”
- It is a statement about base rates. Most tested changes do little, and measurement errors are common, so a very large result is drawn from a population where errors outnumber transformative effects. The law does not say large effects never occur; it says the burden of proof should scale with the size of the claim.
- “A very small p-value means the result is not an artefact.”
- A tracking bug produces a highly significant result, because the spurious difference is systematic rather than random. Significance measures how unlikely the data is under the null, not whether the data was collected correctly. A tiny p-value on an implausible effect makes an instrumentation problem more likely rather than less.
- “The checks are worth running only on results that look too good.”
- Surprising losses deserve identical scrutiny and rarely get it, because a bad result feels self-explanatory. Investigating only implausible wins means errors that inflate results get caught while errors that suppress them do not, which biases everything that ships.
Frequently asked questions
What should I check when a result looks too good?
In order of how often they find something: the sample ratio, since a broken split invalidates everything; whether the metric is computed and fired identically in both arms, which catches duplicate or missing events; whether the effect is concentrated in one segment or time window, which points at an outage or bot traffic; and whether the analysis unit matches the randomisation unit. All four take well under an hour.
How large does an effect have to be before it is suspicious?
It depends on your own distribution of past results rather than on a universal threshold. If your typical win is 1 to 3% and the best result of the last two years was 8%, then anything in double figures deserves scrutiny. Keeping a record of historical effect sizes gives you a calibrated prior, which is more useful than any rule of thumb someone else's programme would supply.
Do genuinely large effects ever happen?
Yes — on entirely new surfaces, on flows that were badly broken, and where a change removes a real obstacle rather than adjusting a working experience. Fixing a checkout step that silently failed on a common browser can produce an enormous measured lift, and correctly so. The point of the law is not to disbelieve those but to establish which kind of result you have before acting on it.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.