A multivariate test varies several elements simultaneously and measures every combination, so it can estimate how those elements interact as well as their individual effects. That interaction estimate is the reason it exists and the reason it needs so much traffic.
A multivariate test varies more than one element at a time. Two headlines crossed with two images crossed with two button colours produces eight combinations, and every user sees one of them. Its purpose is not merely to test eight designs — an A/B/n test does that more cheaply — but to separate the effect of each element from the effects of the others, and to measure how they interact.
That separation is a genuine capability. A full factorial design lets you estimate the headline's effect averaged across both images, the image's effect averaged across both headlines, and crucially the interaction between them: whether headline B works better with image X than with image Y. No sequence of A/B tests recovers that, because each test holds the other elements fixed at one setting and cannot see what happens at the others.
The cost is that interactions are expensive to estimate. An interaction is a difference of differences, so four separate estimates each contribute their noise, and detecting one reliably takes roughly four times the sample of a main effect of the same size. With eight arms splitting the traffic as well, a 2 × 2 × 2 test designed to detect interactions can easily need an order of magnitude more traffic than the equivalent A/B test. That arithmetic is why multivariate testing has a reputation for being impractical, and the reputation is largely earned.
The honest question before running one is whether you actually need the interaction. Most product teams do not: they want to know which of several designs performs best, and the elements plausibly do not interact much. In that case an A/B/n test on the handful of complete designs worth considering is faster and needs far less traffic. Multivariate earns its place when the elements genuinely might conflict — two things competing for the same attention, or a change whose effect you suspect depends on another change shipping at the same time.
There is a middle option worth knowing. Fractional factorial designs test a deliberately chosen subset of combinations, estimating main effects and selected interactions while leaving others confounded with each other. That cuts the arm count substantially — a 2⁴ design reduced from sixteen arms to eight — at the price of not being able to distinguish certain interactions. For screening which of many elements matter at all, it is often the right tool.
The arm count and the sample requirement are what decide feasibility, and both grow faster than intuition suggests.
arms = Π (levels of each factor)Three factors at two levels each gives eight. Adding a fourth doubles it to sixteen.
Y = β₀ + Σ βᵢ·factorᵢ + Σ βᵢⱼ·(factorᵢ × factorⱼ) + …Main effects plus interaction terms. The interaction terms are what an A/B/n test cannot supply.
n_interaction ≈ 4 × n_main effectA difference of differences: four noisy estimates combined — see interaction effect.
n_arm = N / armsEight arms means an eighth each, on top of the fourfold interaction penalty — see the ANOVA calculator.
A landing page team wants to test two headlines, two hero images and two call-to-action labels. Baseline conversion is 5.4% and they have 400,000 visitors available over a month. They compare a full factorial design against the alternative of an A/B/n on the three most promising complete designs.
The factorial design can resolve main effects of 6.8% and interactions only above 13.6%. The A/B/n resolves 4.9% on the designs that matter.
The interaction row is where the decision sits. A 13.6% relative interaction is enormous — it would mean one headline works dramatically better with one image than the other — and effects that large between page elements are uncommon. So this factorial design is very likely to spend a month establishing that no interaction is detectable, which is a weak conclusion rather than an informative one. The A/B/n alternative uses the same traffic to resolve 4.9% effects on the three designs anyone would actually ship. The case for going factorial anyway is if there is a specific reason to expect conflict — say the headline and the call-to-action both make an urgency claim, and running both might read as pushy. That is a real hypothesis about an interaction, and it is worth the traffic. Absent such a reason, the multivariate design is buying a capability the team has no use for.
In this blog, we’ll explain why running multiple experiments simultaneously is not only feasible but also beneficial, explore its key advantages and potential challenges, and share best practices for successful implementation.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.