A treatment group is the set of users who receive the change being tested. Its outcome is compared against the control group's, and the difference between them is the effect — provided every user assigned to it is counted whether or not they actually experienced the change.
The treatment group receives the change; the control group does not; the difference in their outcomes is the effect. "Variant" is the same thing in platform vocabulary, and multi-arm tests simply have several — variant A, variant B — each compared against the same control. Nothing about the word choice changes the statistics.
The rule that does the real work is who gets counted. Every user assigned to the treatment group belongs in the analysis, including those who never reached the changed surface, never opened the app, or bounced immediately. This is analysis by intent to treat, and it is what preserves the randomisation: the groups were formed at random, so they are comparable, and dropping people from one arm on the basis of their behaviour destroys exactly that property.
The temptation to drop them is strong because they dilute the effect. If a change touches only the 15% of users who reach a particular page, then 85% of the treatment group is identical to control and the measured effect is roughly a seventh of the real one. That looks like waste, and the naive fix — analyse only those who saw it — is invalid, because reaching the page may itself be influenced by the treatment, so the two arms would be filtered on different populations.
The valid version is triggered analysis, and the distinction is precise: the trigger must be an event that occurs *before* the treatment could influence it, and it must be recorded identically in both arms. "Reached the checkout page" is usually a valid trigger, because the change happens on that page. "Clicked the new button" is never valid, because the button does not exist in control. Getting this wrong is one of the more common ways an experiment produces a large, confident and completely spurious effect.
The other counting question is what happens when a treatment user cannot receive the change — a feature flag fails, an old client version cannot render it, a request errors. They stay in the treatment group. Removing them selects on something that may correlate with the outcome, and if failures are more common on slow devices then removing them quietly compares fast-device treatment against all-device control.
Two estimators, and the relationship between them. The dilution factor is what makes triggering attractive and the exclusion restriction is what makes it legitimate.
ITT = E[ Y | assigned treatment ] − E[ Y | assigned control ]Counts everyone as assigned. Always unbiased, because it compares the groups randomisation created.
ITT = effect_on_exposed × exposure rateA change touching 15% of users shows about a seventh of its true effect when measured across everyone.
restrict both arms on an event the treatment cannot influenceReaching a page is usually fine. Interacting with the new element never is, since control has no equivalent.
n_triggered ≈ n_all × exposure rateTriggering on a 15% surface cuts the required traffic by roughly the same factor — see the sample size calculator.
A change to the returns policy page is tested on 240,000 assigned users per arm. Only 9% of users ever visit that page. Three analyses are compared: everyone assigned, everyone who visited the page, and everyone who clicked the new explanatory link.
The first two analyses are the same finding at different dilutions. The third is not a finding at all.
All-assigned and triggered agree once dilution is accounted for: 0.46 × 0.09 = 0.041, which is the 0.04 the full-population analysis reports. That consistency is the check worth running, and it confirms the triggered analysis is measuring the same effect with less noise rather than a different one. The click-based comparison is the trap. Its +3.90 pp is enormous and meaningless, because the 6,140 people who clicked a new link are self-selected — they were more engaged with returns policy before they clicked anything, and control contains no equivalent group to compare them against. There is no way to construct one after the fact. Note also that page visits were near-identical between arms, 21,600 against 21,480, which is what makes the trigger valid here: had the treatment changed who reached the page, even the triggered analysis would have been comparing different populations.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.