A control group is the set of users kept on the existing experience so the treated group has something to be compared against. It supplies the counterfactual — what would have happened anyway — which is the only thing that turns a measured change into a measured effect.
An experiment measures a difference, and a difference needs two things to compare. The control group is the second one: users who continue to receive the existing experience while the treated group receives the change. Without it you can observe that conversion was 4.1% after a launch and you cannot say what it would have been otherwise, which is the only question that matters.
That is why a before-and-after comparison is not an experiment. Between last month and this one, the season changed, a competitor ran a promotion, the marketing mix shifted and the product got three other releases. Attributing the difference to your change requires assuming none of that mattered. A concurrent control group experiences all of it too, so everything except the treatment cancels — which is what randomisation buys and what makes the comparison causal rather than merely chronological.
The requirement is that control and treatment differ in exactly one respect. Randomization is what delivers that for everything you did not think to measure, which is the majority of what matters about a user. It also means the two groups must be handled identically in every other way: assigned at the same moment, subject to the same filters, measured with the same instrumentation. A control group defined by a different query, or filtered after assignment on something the treatment influenced, has stopped being a valid baseline.
Contamination is the usual failure and it is quiet. If control users can see the treatment — through a shared account, a public URL, a colleague's screen, or a marketplace where both sides interact — the difference between arms shrinks and the effect is understated. In two-sided markets this is severe enough to need a different design entirely: a switchback test or market-level randomisation rather than user-level. The absence of an obvious error message is not evidence that contamination did not happen.
Two related shapes are worth distinguishing. A holdout group is a control kept back after the decision, to keep measuring something already shipped. A global holdout is a permanent one, receiving nothing, which is what makes it possible to measure the accumulated effect of a year of releases rather than trusting the sum of individual test results.
The estimator is a subtraction. What earns the causal interpretation is not the arithmetic but how the two groups were formed.
effect = E[ Y | treated ] − E[ Y | control ]Causal only when assignment was random. Otherwise it is a difference between two populations that also differ in other ways.
E[ Y | control ] ≈ E[ Y(0) ]The counterfactual: what the treated group would have done untreated. The whole reason the group exists — see counterfactual reasoning.
Y_after − Y_before = effect + trend + seasonality + everything elseA concurrent control absorbs every term except the effect, because it experienced them too.
observed effect = true effect × ( 1 − contamination rate )Attenuates towards zero, so contamination hides real effects rather than inventing them — see the sample size calculator.
A team ships a new pricing page to everyone and compares the four weeks after against the four weeks before, then re-runs the same question properly with a concurrent 50/50 control on the following release.
The before-and-after comparison reports +12.8%. The randomised comparison reports +2.1%, most of which is not distinguishable from zero.
The control arm is the whole story. It moved from 3.82% to 4.22% without receiving anything — a 10.5% lift driven by seasonality and whatever else happened that month — and the before-and-after design books every bit of that to the pricing page. The randomised comparison sees both arms experiencing the same month and isolates the 2.1% that the change is actually responsible for. Two things follow. The honest reading of the experiment is not "the change worked" but "if there is an effect it is small, and this test could not resolve it" — the interval spans zero. And the £-value a business case would have been built on was six times too large, which is the ordinary magnitude of this error rather than an extreme one. Any measurement of a change shipped to everyone at once is subject to it, and no amount of statistical care afterwards recovers the counterfactual that was never collected.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
This guide aims to give you practical principles to develop A/B testing mindset. Principles that will help you survive, thrive, and systematically reach the 20-30% test success rate that creates dramatic impact in your bottom line.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.