What is A/B testing?

An A/B test splits incoming users at random between a control experience and one or more variants, then compares an outcome metric across the groups. The mechanism that makes this worth doing is randomization, and it is doing more work than it appears to. Because assignment is random, the two groups are on average identical in every respect — including the ones you never measured and the ones you have never thought of — so any systematic difference in outcomes has to have come from the change. This is why an A/B test yields a causal estimate while a before-and-after comparison does not.

Everything else in the discipline is scaffolding around that one idea. You fix a primary metric before starting, so the result cannot be chosen after the fact. You compute a sample size from the smallest lift worth shipping, so the test has the statistical power to answer the question. You run it for whole weeks, because weekday and weekend users differ. And you check that traffic actually split the way you asked, because a sample ratio mismatch means the randomisation you are relying on did not happen.

The failures worth knowing are mostly failures of the assumption rather than of the arithmetic. Interference: if users in one arm affect users in the other — a marketplace where a variant consumes shared inventory, a social product where content crosses arms — the groups are no longer independent and the estimate is biased. Novelty and primacy: a visible change gets a temporary reaction that fades, so a two-week test on a redesign measures curiosity as much as value. And dilution, where the analysis includes users who never reached the surface you changed, which shrinks a real effect towards nothing.

There are also questions an A/B test simply cannot answer, and reaching for one anyway is the more expensive mistake. Anything you cannot randomise at the user level — a brand campaign, a pricing change with fairness constraints, a TV flight — needs a different design, usually a geo experiment that randomises regions instead. Anything whose effect accrues over months exceeds the horizon of a test you will actually wait for. And the cross-channel question — what did all our marketing do together — is what marketing mix modelling exists for.

Used well, the value is less about individual wins than about the rate of learning. A programme that runs properly powered tests, records the negatives as carefully as the positives, and resists the urge to slice a flat result until something appears will accumulate a genuine picture of what moves its metrics. A programme that runs underpowered tests and reports whichever segment came out green will accumulate confident nonsense, and will do it faster.

You want to test a redesigned product page. Conversion is 3.2%, and the smallest lift worth the engineering cost is 5% relative — 3.2% to 3.36%. You have 45,000 sessions a day to split between two arms and want the conventional 5% two-sided level at 80% power.

The rounding is not padding. Conversion differs systematically between weekdays and weekends, and a test that runs 8 days contains an unequal mix — which does not bias the comparison, since both arms see the same days, but does inflate the variance and can make a genuine effect harder to detect. Two whole weeks also gives novelty effects room to decay, so you can compare the first week's lift against the second and see whether the effect is stable. If it is not, the number to trust is the later one. And before reading anything, check the split: 22,500 against 22,500 should be within a few hundred, and a gap larger than chance allows means the randomisation is broken and the estimate is not causal.