When allocation point and exposure point differ
A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.
What is intent-to-treat analysis?
Intent-to-treat analyses everyone by the arm they were assigned to, regardless of what they actually received or whether they engaged. It preserves the randomisation, which is the only thing that makes the comparison causal, at the cost of diluting the measured effect.
ITT
In plain English
Intent-to-treat is a rule about who goes in the analysis: everyone, in the arm they were assigned to, whatever happened next. Users who never opened the app, never reached the changed page, or hit an error that prevented the feature loading all stay in the treatment group. It is the default in clinical trials for good reason and it is the correct default in product experimentation for the same reason.
The reason is that randomisation is the only thing making the two groups comparable, and it applies to assignment rather than to what followed. Once you start removing users on the basis of their behaviour, the groups are no longer the ones randomisation created — they are groups selected on something that may itself depend on the treatment. That is the whole guarantee, and it is discarded the moment an exclusion is made after the fact.
The alternative that tempts everyone is per-protocol analysis: compare only those who actually received the treatment against control. It sounds like it isolates the effect and it reliably inflates it. Users who engaged with a new feature are more engaged in general, and control contains no equivalent self-selected group to compare against — nor can one be constructed after the fact, because the feature does not exist there to select on.
The honest cost of ITT is dilution. If only 15% of assigned users ever reach the changed surface, then 85% of the treatment arm is identical to control and the measured effect is roughly a seventh of the effect on those who encountered it. The estimate is still unbiased for the population effect — it is simply answering "what happens if we deploy this" rather than "what does it do to someone who sees it", and those are different questions with different right answers.
Two legitimate refinements recover sensitivity without breaking the randomisation. Triggered analysis restricts both arms on a pre-treatment event — reaching the page — which is valid provided the trigger cannot be influenced by the treatment; see the exposure point. And the LATE divides the ITT effect by the difference in take-up to recover the effect among those the assignment actually moved. Neither drops users from one arm on the basis of their behaviour, which is the line that must not be crossed.
The formula
The estimand, its dilution, and the two valid ways of recovering the effect on those who were actually reached.
- The estimand
ITT = E[ Y | assigned treatment ] − E[ Y | assigned control ]Assigned, not treated. Unbiased because it compares exactly the groups randomisation created.
- The dilution
ITT = effect_on_reached × reach rateA surface reached by 15% of users shows about a seventh of the effect across everyone assigned.
- Recovering it — instrumental variables
LATE = ITT / ( take-up_treated − take-up_control )Scales the diluted estimate back up to the compliers — see LATE.
- Recovering it — triggering
restrict both arms on a pre-treatment eventValid only if the trigger cannot respond to the treatment — see the sample size calculator.
Worked example
A feature is assigned to 120,000 users per arm but lives behind a settings page only some users visit. Three analyses are compared: intent-to-treat, a per-protocol comparison of users who enabled the feature against all control users, and a triggered analysis on settings-page visitors.
ITT and the triggered analysis agree once dilution is accounted for. The per-protocol figure is seven times larger and is not a causal estimate.
The consistency check works: 0.94 × 0.119 = 0.112, which matches the ITT's 0.11 almost exactly, confirming that the triggered analysis is measuring the same effect with less noise rather than a different one. The per-protocol number is the trap. Those 5,940 users chose to enable a retention-relevant feature, which means they were more committed before they clicked anything, and there is no comparable group in control to set against them — control users cannot enable a feature that does not exist for them. The 7.8 points is mostly who those people were. Note also what makes the triggered analysis legitimate here: settings-page visits are near-identical across arms, 14,300 against 14,180, so the treatment is not changing who qualifies. Had the feature's presence drawn more people into settings, even that analysis would have been comparing different populations.
Common misconceptions
- “Users who never saw the feature should be excluded — they only dilute the result.”
- They dilute it and they preserve validity, which is the more important property. Excluding on post-assignment behaviour destroys the randomisation, because whether someone reached the feature may itself depend on the treatment. Triggering on a pre-treatment event recovers the sensitivity without that cost.
- “Comparing users who engaged against control shows what the feature does.”
- It compares self-selected engaged users against an unselected group, so it measures engagement as much as the feature. Control contains no equivalent set and one cannot be constructed, because the feature does not exist there to select on. This comparison reliably produces large effects that do not survive a proper test.
- “Intent-to-treat understates the effect, so it is the conservative choice rather than the correct one.”
- It is unbiased for the question it answers — what happens if you deploy this to the assigned population. That is often exactly the decision being made. It understates the effect on those who were reached, which is a different estimand, and the valid routes to that are triggering or an instrumental-variables estimate rather than dropping users.
Frequently asked questions
Why keep users who never experienced the treatment?
Because randomisation only guarantees that the assigned groups are comparable, and any exclusion based on what happened afterwards breaks that guarantee. Whether someone reached the feature can itself depend on the treatment, so filtering on it compares two differently selected populations. Keeping everyone costs sensitivity and preserves the only property that makes the comparison causal.
How do I recover sensitivity without breaking the randomisation?
Trigger on a pre-treatment event recorded identically in both arms — reaching the page whose content changed is usually valid, and interacting with the new element never is. Or use the instrumental-variables route and divide the ITT effect by the difference in take-up between arms, which gives the effect among compliers. Both keep the arms comparable; dropping non-engagers does not.
What about users where the feature failed to load?
Keep them in the treatment arm. Delivery failures correlate with device, connection and client version, all of which relate to the outcome, so removing them leaves a healthier treatment group compared against an unfiltered control. The failure rate is worth investigating in its own right as a finding about the implementation, but it is not a data-quality exclusion.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.