A switchback test randomises time periods rather than users, applying one condition to everyone during each interval and alternating between them. It is the design for marketplaces and shared systems, where users in different arms would otherwise affect each other.
User-level randomisation assumes one user's assignment does not affect another's outcome. In a marketplace that assumption fails immediately. If a pricing algorithm is shown to half the riders, they compete with the other half for the same drivers, and a change that wins by capturing supply looks good while creating nothing. The arms are not independent, so the comparison is not measuring what it claims.
A switchback test removes the problem by making the randomisation unit time. The entire system runs condition A for thirty minutes, then condition B for thirty minutes, then A again — with the order randomised rather than strictly alternating. During any interval everyone experiences the same condition, so there is no cross-arm competition to distort things. The comparison is between periods rather than between people.
The price is a dramatic reduction in sample size, and it is the fact that governs everything about the design. A week of thirty-minute intervals is 336 units, not the hundreds of thousands a user-randomised test would have. Statistical power comes from the number of switches, so switchback tests are structurally underpowered relative to A/B tests and need to run for weeks to detect effects that a normal test would resolve in days.
Choosing the interval length is the main design decision and it is a genuine trade. Short intervals give more units and therefore more power, and they increase carryover — the effect of one period bleeding into the next, since a driver positioned by condition A is still positioned at the start of condition B. Long intervals reduce carryover and leave you with very few units. The usual approach is to set the interval from how long the system takes to settle after a change, then discard a burn-in window at the start of each period from the analysis.
Two further points matter in practice. Time is not exchangeable — Friday evening is not Tuesday morning — so the randomisation must be balanced across time-of-day and day-of-week, and the analysis should account for those as covariates. And because periods adjacent in time are correlated, the standard error needs to reflect that: treating 336 intervals as independent observations overstates precision, and a block bootstrap or a model with autocorrelated errors is the standard correction.
The unit is an interval rather than a user, and every consequence for power and for variance estimation follows from that.
units = duration / interval lengthA week of 30-minute intervals gives 336. This is the sample size, not the number of users.
effect = mean( Y | periods assigned A ) − mean( Y | periods assigned B )A difference between time periods. Balance the assignment across day-of-week and hour, or the comparison is confounded with time.
discard the first w minutes of each intervalThe system is still settling from the previous condition. w is set by how long the market takes to re-equilibrate.
use a block bootstrap or autocorrelation-robust standard errorsAdjacent intervals are not independent, so naive standard errors are too small — see bootstrap.
A delivery marketplace tests a new courier-matching algorithm. User-level randomisation is impossible because both arms would draw from the same courier pool. They run a switchback over three weeks with one-hour intervals, discarding the first ten minutes of each as burn-in.
The new algorithm cuts delivery time by 1.6 minutes. With correct standard errors that is p = 0.031, not the p < 0.001 the naive calculation claims.
The two standard errors differ by a factor of 2.4, and only the larger one is defensible. Adjacent hours are strongly correlated — demand, weather and courier supply all persist across an hour boundary — so treating 504 intervals as 504 independent observations claims far more information than three weeks of operation contains. The result survives the correction here, which is fortunate rather than guaranteed; a smaller effect would have crossed from significant to not. Two design notes worth carrying forward. The 17% of data discarded as burn-in is a real cost and the right trade: without it, every interval's first minutes are contaminated by the previous condition, biasing the comparison towards no effect. And three weeks was necessary rather than cautious — at 504 units this design has roughly the power of a 500-observation experiment, which is why switchbacks are slow even on high-volume systems.
Our “Geo Testing: Unlocking True Incrementality” webinar explored how teams can measure real-world impact when A/B testing isn’t possible - using geographic experiments and synthetic controls to reveal true incremental lift in marketing and product initiatives.
Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.