What is concurrent testing?

Any organisation running experimentation at scale runs many tests at once on overlapping users. The instinct that this must contaminate the results is understandable and mostly wrong. Provided each experiment randomises independently, the second test's variants are spread evenly across the first test's arms, so its effect appears as noise in both — which inflates variance slightly and does not bias the estimate.

The word doing the work there is independently. If assignment is computed by hashing the user id alone, without the experiment id, every user lands in the same relative position in every test, and the tests share populations rather than crossing them. Then one experiment's treatment is systematically present in another's treatment arm, and the effects genuinely are confounded. This is a bucketing failure rather than a property of concurrency, and it is the one thing that must be right before overlap is safe.

What concurrency genuinely cannot give you is the combined effect. Each experiment measures its change averaged over whatever the other experiments were doing, so two tests that each report a win tell you nothing about what happens when both ship. If they touch the same surface they may well conflict — an interaction effect — and the roadmap's assumption that gains add up is exactly the assumption concurrency leaves untested.

The practical response is proportionate rather than absolute. Most pairs of experiments are unrelated: a checkout change and an email subject line do not interact, and treating every overlap as a hazard would reduce an experimentation programme to a queue. What deserves attention is overlap on the same surface or the same decision point, which is both where interactions are plausible and where two teams are most likely to be working simultaneously without noticing.

So the useful discipline is a register rather than a serialisation policy: record what is running where, make overlaps on a shared surface visible, and sequence those deliberately or test them together as a factorial. And when a shipped combination underperforms the sum of its parts, treat that as the interaction evidence it is — that gap usually arrives after the fact, and it is the main way an organisation learns which of its surfaces are crowded.

A team runs six concurrent experiments across the same user base. Two of them — a trust-badge row and an urgency banner — both change the product page above the fold. The other four touch email, search ranking, the mobile navigation and a billing screen. All six report significant wins and all six ship in the same release.

This is concurrency working as designed and being misread at the reporting stage. The individual experiments were all correctly randomised and correctly analysed — none was contaminated, and the four unrelated ones summed cleanly. The shortfall traces almost entirely to the pair sharing a surface, which the follow-up factorial confirms at −0.39 of the 0.43. The lesson is not to stop running tests in parallel; it is that the register should have flagged two experiments on the same region of the same page, and those two should have been sequenced or run as a factorial from the start. Worth noting how cheap the diagnosis was after the fact: knowing which pair to suspect made it a single 2 × 2 rather than an audit of all fifteen possible pairings. That is the practical value of recording what is running where.