When allocation point and exposure point differ
A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.
What is selection bias?
Selection bias is any distortion caused by how units entered your data. If the people you observe differ systematically from the ones you want to draw conclusions about, the estimate is wrong in a direction the sample size will never fix.
In plain English
Selection bias is a failure of who, not of how much. Somewhere between the population you want to describe and the rows in your table, a filter operated that was related to the outcome — and once that has happened, no amount of additional data helps. Collecting ten times as many biased observations gives you a very precise answer to a question you did not ask. This is what separates bias from noise, and why it deserves more attention than it usually gets.
It arrives in several recognisable shapes. Survivorship bias analyses only the units that lasted: churned customers are missing from a retention study, failed campaigns from a case-study library, and the answer to "what do our best accounts have in common" is silently conditioned on being a best account. Non-response bias affects every survey, because the people who reply to a satisfaction survey are not a random sample of customers — the delighted and the furious both answer, and the vast indifferent middle does not. Self-selection affects every opt-in feature: beta users are enthusiasts, so a beta's engagement numbers describe enthusiasts.
The version that quietly ruins experiments is post-assignment filtering. Randomization gives you two comparable groups, and any filter applied *after* assignment that operates differently across arms takes that away. Excluding users who did not complete onboarding, when the variant changed onboarding. Dropping sessions shorter than five seconds, when the variant loads more slowly. Removing suspected bot traffic with a heuristic that trips more on one experience. Each of these looks like data hygiene and each re-introduces selection into a design built specifically to eliminate it — which is why a sample ratio mismatch check must run before anything else.
It is distinct from confounding even though the two travel together. A confounder is a common cause of treatment and outcome, distorting a relationship between variables you have. Selection bias is about which rows exist at all, distorting the population you are describing. The consequence is that controlling for variables — the standard answer to confounding — often does nothing here, and can make things worse when the selection variable is a collider.
The defences are mostly design rather than analysis. Define the population before you filter, and apply identical filters to every arm. Analyse by intention to treat: everyone assigned stays in the analysis, whatever they subsequently did. Compare respondents against non-respondents on whatever you know about both. And when a filter is genuinely necessary, evaluate it on a pre-experiment window so it cannot be affected by the treatment.
The formula
Selection bias is a conditioning problem, and stating it that way makes clear why more data does not help.
- The general form
E[Y | S = 1] ≠ E[Y]S is the indicator for being selected into the data. Bias exists whenever selection is related to the outcome, however the selection happened.
- Why sample size cannot fix it
bias = E[Y | S = 1] − E[Y], independent of nNoise falls as 1/√n; this term does not move. A large biased sample is a precisely wrong answer.
- Post-assignment filtering
( Y(1), Y(0) ) ⫫ D but not ( Y(1), Y(0) ) ⫫ D | SRandomisation guarantees the first. Conditioning on anything measured after assignment can destroy it, which is why intention-to-treat is the default analysis.
- Non-response weighting
Ŷ = Σ wᵢ·Yᵢ / Σ wᵢ, wᵢ = 1 / P(respond | Xᵢ)Corrects for selection on measured characteristics only. It assumes non-response is unrelated to the outcome given X, which is exactly what you cannot check.
Worked example
A team tests a longer, more detailed onboarding flow. To compare like with like they analyse only users who completed onboarding in both arms. Among completers, 30-day retention is 61% in the variant against 55% in control — a clear win, p < 0.001.
Analysed on completers the variant wins by six points. Analysed on everyone assigned it loses by three.
Both numbers are correctly computed and only one of them answers a question anyone should act on. The longer flow drove 6,500 users out before completion, and the ones who dropped were disproportionately the marginal, less committed users — so the variant's completer group is a more motivated population than control's, and it retains better for that reason alone. Conditioning on completion is conditioning on a collider: it is caused by the treatment and it predicts the outcome. The intention-to-treat comparison keeps everyone assigned in their original arm and shows the flow is a net loss. As a rule, if a filter can be affected by the treatment, applying it forfeits the randomisation you paid for.
Common misconceptions
- “Our sample is large, so it is representative.”
- Size and representativeness are unrelated. A million responses from people who chose to respond describe people who choose to respond, very precisely. The classic demonstration is the 1936 Literary Digest poll, which called the US presidential election wrongly on 2.4 million responses because of who was on its mailing list.
- “We removed outliers and bot traffic to clean the data, which improves the estimate.”
- Only if the removal rule is applied identically across arms and cannot be affected by the treatment. A filter that trips more often on one experience re-introduces selection into a randomised design. Define the rule in advance, evaluate it on pre-assignment data where possible, and check the arm counts afterwards.
- “Analysing only users who actually used the feature gives a cleaner estimate.”
- It gives an estimate for a self-selected population — the ones who chose to use it — which is usually the most engaged group and is not comparable to the whole control arm. Keep everyone assigned in the analysis, and if you genuinely need the effect on users who took it up, use an instrumental-variable estimator rather than a filter.
Frequently asked questions
What are the main types of selection bias?
Sampling bias, where the frame you drew from differs from the population. Self-selection, where units chose whether to be included — every opt-in beta and voluntary survey. Survivorship, where only units that lasted are visible. Non-response, where who answers is related to what they would have said. And attrition, where units drop out of a study differentially. They share one signature: the filter is related to the outcome.
How do I correct for selection bias after the fact?
Only partially, and only for selection on things you measured. Weighting by the inverse probability of being included, or reweighting the sample to match known population margins, both help — and both assume that within levels of the measured variables, selection is unrelated to the outcome. That assumption is untestable and is exactly what fails in the hard cases. Prevention through design is worth far more than any correction.
Can a randomised experiment still have selection bias?
Yes, in two places. Who entered the experiment at all — a test run only on logged-in users tells you about logged-in users. And anything applied after assignment: filters, exclusions, and differential dropout can all destroy the balance randomisation created. The first limits generalisability, the second invalidates the comparison, and the second is the one that produces confidently wrong numbers.
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.