What is an interaction effect?

Most experimentation programmes assume effects add up. Team A lifts conversion by 2%, team B lifts it by 3%, and the roadmap quietly promises 5%. An interaction is what happens when that arithmetic fails — when the effect of one change is different depending on whether the other is present. The combined result might be 5%, or 1%, or −2%, and nothing in either team's individual readout distinguishes those cases.

The mechanism is usually mundane rather than exotic. Two changes competing for the same screen space, so each dilutes the other's prominence. Two prompts targeting the same moment in the funnel, where the second is redundant once the first has done its work. A discount banner and a free-shipping banner that between them make the site look like a clearance sale. In each case both changes are genuinely good in isolation and the pair is worse than either — a pattern common enough that "we shipped both wins and the metric did not move" is a familiar experience in mature programmes.

Detecting an interaction requires a design that lets both changes vary. A 2 × 2 factorial runs four arms — neither change, A only, B only, both — and the interaction term measures how far the both-arm outcome departs from what the two individual effects predict. This is what a multivariate test is really doing, and its reputation for needing enormous traffic is deserved: detecting an interaction reliably takes roughly four times the sample needed to detect a main effect of the same size, because the estimate is a difference of differences and the noise compounds at each step.

That cost is why most organisations run concurrent tests instead and simply accept the risk. Two experiments overlap on the same users, each randomised independently, and the assumption is that interactions are rare enough to ignore. Usually they are — an interaction needs the two changes to touch the same surface or the same decision, and most pairs of experiments do not. The assumption becomes dangerous exactly when it is most likely to be wrong: two teams working on the same funnel step at the same time is precisely the situation where an interaction is plausible and concurrent testing is blind to it.

Two practical habits are worth more here than any test. Keep a register of what is running where, so overlaps on the same surface are visible and can be deliberately sequenced rather than accidentally combined. And when a shipped combination underperforms its predicted sum, treat that as the finding it is rather than as noise — it is the clearest evidence an interaction exists, and it usually arrives after the fact. We work through the wider question in parallel testing in A/B testing.

Two teams test simultaneously on the product page. One adds a trust-badge row; the other adds an urgency banner. Both are run as independent A/B tests on overlapping traffic, and both report wins. A 2 × 2 factorial is run afterwards on 60,000 users per cell to check the combination before shipping both.

This is the failure mode the additive assumption hides. Two honest wins, two correctly analysed experiments, and shipping both costs 0.35 percentage points against what was forecast — on a page where that is real money. The mechanism is not mysterious once seen: both changes compete for attention in the same region above the fold, and a page carrying trust badges and an urgency banner reads as harder-selling than either alone. Note the sample size it took to establish this. Each main effect was detectable at around 15,000 users per arm; the interaction needed 60,000 per cell, four times as much, which is exactly the multiplier the formula predicts and the reason this check is usually skipped. The cheaper habit is not to run the factorial every time but to notice which experiments touch the same surface and sequence those deliberately.