The Ultimate Guide to Choosing Your Perfect MMM Tool
Robyn, Meridian, Orbit, or custom-made MMM? In this blog post, we compare leading free, paid or custom MMM tools, to help you decide which best fits your needs.
What is overfitting?
Overfitting is when a model learns the noise in its training data rather than the underlying pattern. It fits the data it was built on unusually well and performs badly on anything new, because much of what it learned was specific to that sample.
In plain English
Every dataset contains signal and noise. A model with enough flexibility will fit both, because it has no way to tell them apart — the noise is, from the model's perspective, just more pattern to reproduce. The result looks superb: residuals small, R-squared high, the fitted line passing near every point. Then new data arrives, containing different noise, and the model's performance collapses. That gap between how well a model fits what it has seen and how well it predicts what it has not is overfitting.
The underlying trade-off is between bias and variance. A very simple model is biased — it cannot represent the real relationship, so it is wrong in a consistent way. A very flexible model has high variance — it would produce a noticeably different answer if refitted on another sample from the same population. Total error is the sum of both, so there is an optimum in the middle, and the whole discipline of model selection is a search for it.
In practice overfitting comes from a few recognisable situations. Too many parameters relative to observations, which is chronic in marketing mix modelling, where two years of weekly data gives about a hundred rows for a dozen or more effects. Choosing a specification by trying many and keeping the best fit, which fits noise through the selection process even if each candidate model is simple. And tuning on the same data used to evaluate, which quietly converts a holdout into training data.
Detection has one reliable method: evaluate on data the model has never seen. Split the sample, hold out a test set, or use cross-validation — and for anything with a time dimension, split by time rather than at random, since randomly held-out weeks are surrounded by weeks the model has seen and are therefore far too easy to predict. A large gap between training and held-out performance is the definition of the problem, and no in-sample statistic substitutes for it. Adjusted R-squared penalises parameters but not the search process that chose them.
The remedies all reduce effective flexibility. Fewer parameters. Regularisation, which shrinks coefficients towards zero and is the standard answer when predictors are many and correlated. Informative priors in a Bayesian model, which do the same thing more legibly. Bringing in more data, when it is available. And, most reliably in a business setting, imposing structure from domain knowledge: an adstock decay and a saturation curve are strong constraints that let a media model use a hundred observations sensibly, precisely because they rule out shapes nobody believes.
The formula
The decomposition explains why the optimum is in the middle, and the criteria below are attempts to find it without a holdout.
- Bias-variance decomposition
E[(y − f̂(x))²] = Bias² + Variance + σ²_irreducibleFlexibility lowers bias and raises variance. The third term is noise no model can remove, and it sets the floor on achievable error.
- Akaike information criterion
AIC = 2k − 2·ln(L̂)Penalises parameters. Lower is better, and only comparable across models fitted on identical data.
- Bayesian information criterion
BIC = k·ln(n) − 2·ln(L̂)Penalises more heavily as n grows, so it selects simpler models than AIC. Preferable when the goal is a model you will interpret rather than only predict from.
- Ridge regularisation
minimise Σ(yᵢ − ŷᵢ)² + λ·Σβⱼ²The penalty shrinks coefficients, trading a little bias for a large reduction in variance. λ is chosen by cross-validation, on training data only.
Worked example
An analyst builds a weekly sales model on three years of data. Wanting the best fit, they add channel interactions, a promotion flag per retailer, and a separate coefficient for each week of the year. In-sample R-squared reaches 0.97. The model is then evaluated on the following six months.
Held-out error is six times the in-sample error, and worse than a baseline containing nothing but a trend and monthly seasonality.
Seventy-one parameters on 156 observations is roughly one parameter for every two weeks, and at that ratio a model can reproduce almost any history without learning anything about the future. The fifty-two week-of-year coefficients are the clearest offender: each is estimated from three observations, so each is largely memorising three specific weeks. The decisive comparison is against the baseline — a model that loses to a trend and month dummies is not a weak model, it is a harmful one, because its coefficients will be quoted in budget meetings. The fix here is structural rather than statistical: replace the 52 dummies with a smooth seasonal term, drop the interactions the data cannot support, and impose adstock and saturation shapes that encode what is already known.
Common misconceptions
- “A model with a high R-squared is a good model.”
- In-sample fit rises mechanically with flexibility, so a high value can be evidence of overfitting rather than of quality. The informative comparison is between training and held-out performance, and against a trivial baseline. A model that fits at 0.97 and loses to a seasonal baseline out of sample has learned nothing worth having.
- “More data always fixes overfitting.”
- It helps, and it does not help if the model grows with the data or if the extra observations are highly correlated with existing ones. Fifty-two more weeks of a seasonal series adds far less independent information than the row count suggests. Constraining the model — through regularisation, priors or structure — is usually the more effective lever.
- “We used cross-validation, so the model is not overfitted.”
- Only if the folds respect the structure of the data and the tuning happened inside them. Randomly splitting time series lets the model see weeks either side of every held-out week, which inflates performance badly — split by time instead. And selecting features on the full dataset before cross-validating leaks the test data into training.
Frequently asked questions
How do I know if my model is overfitted?
Evaluate it on data it has never seen and compare against the training performance. A large gap is the definition of overfitting. For time-series data, hold out the most recent period rather than random rows, since random splits let the model interpolate between weeks it has already seen and make almost any model look good. Comparing against a trivial baseline is the other essential check.
What is the best way to prevent overfitting?
Reduce effective flexibility and impose what you already know. Fewer parameters, regularisation, or informative priors all work, and in a business setting structural constraints usually work best — adstock decay and saturation curves rule out shapes nobody believes and let a small dataset go much further. Deciding the specification before looking at the results also removes the selection process as a source of overfitting.
What is underfitting, and how do I avoid overcorrecting?
Underfitting is the opposite failure: a model too simple to capture the real pattern, which performs poorly in-sample and out of sample alike. The distinguishing sign is that both errors are high and similar, whereas overfitting shows low training error and high held-out error. If both are high, add flexibility or better predictors; if they diverge, remove flexibility.
Calculate it
Knowing the term is the easy part
Applying it to a live measurement problem is the part that goes wrong. If you are designing an experiment, reading a result you do not trust, or trying to work out what your marketing actually caused, that is the work we do.