How is an MMM calibrated?

A marketing mix model estimates each channel's contribution from historical variation in spend and outcomes. That is fundamentally a correlational exercise, and it inherits the usual weakness: when two channels moved together, or a channel's budget barely varied, the data cannot separate their effects and the model produces a coefficient anyway. Calibration is the practice of anchoring those estimates to experiments that measured a channel's effect directly.

The problem it fixes is specific. An MMM will report a contribution for every channel in it, and the reported precision reflects how well the model fits rather than how well the effect is identified. Channels whose spend has been flat for two years, or that always move in step with another channel, get coefficients that are essentially assumptions dressed as estimates. Nothing in the output distinguishes those from the well-identified ones.

A geo experiment measures a channel's incrementality causally, which is precisely what the model cannot establish alone. In a Bayesian MMM the natural way to use that is as an informative prior on the relevant coefficient — the experiment says this channel returns around 2.8 with a certain uncertainty, and the model fits everything else subject to that constraint. The experiment pins one part of the model and the model interpolates the rest.

The effect is usually larger than expected because the coefficients are not independent. Constraining one channel to its measured value changes what the model can attribute to the others, since the total is bounded by observed outcomes. Calibrating a single overstated channel typically redistributes contribution across several, which is why one well-chosen experiment can improve an entire model rather than only one line of it.

The practical programme is to calibrate the channels where identification is weakest and the spend is largest, refresh annually because response curves shift, and treat an uncalibrated MMM's channel-level numbers as provisional. This is also what makes the combination of methods coherent rather than a portfolio of competing answers: experiments provide precise causal estimates on a few channels at a time, and the MMM extends that to everything, including channels no experiment can hold out.

An MMM covering five channels is fitted, then recalibrated using two geo experiments — one on branded search, one on television. Neither channel's spend had varied much historically.

The two calibrated channels were the ones the model could least identify — branded search spend had tracked overall demand for years, and television budget had barely moved, so both coefficients were largely assumption. The model had attributed to branded search a great deal of demand that branded search was capturing rather than creating, which is the characteristic error when a channel's spend correlates with demand it does not cause. The row worth noticing is social moving from 11% to 14% without any experiment touching it: constraining two channels changes what the model can attribute elsewhere, so one calibration improves the whole allocation. The fit statistic falling slightly is expected and reassuring rather than a problem — the uncalibrated model fitted the history better precisely because it was free to assign contribution wherever the correlations pointed, and a slightly worse fit against a causally anchored structure is the trade being made deliberately.