Marketing mix model robustness measures whether channel-level results hold up under scrutiny, not how confident the output looks. This matters because flat spend or sparse data can make a channel hard to decompose, even when a model looks confident. Without testing that, marketers trust metrics like MAPE, which can be overfit and miss weak signals. This video explains how perturbing assumptions and comparing priors to posteriors reveals true signal strength.
Why Marketing Mix Model Robustness Requires More Than a Confident Output
Marketing mix model robustness requires more than a confident output, because a model can look polished and still be built on flat spend or sparse data that makes a channel almost impossible to decompose accurately. In this video, Jon Lorenzini, Chief Marketing Science Officer at LiftLab, explains LiftLab’s Channel Signal Strength feature, built to surface a level of transparency he says most modelers don’t offer. Instead of just delivering a confident-looking answer, the feature tests whether that confidence is actually earned.
The method centers on perturbing the model’s own assumptions and watching whether the outputs, the posteriors, snap back to the same story regardless of what was fed in as the prior. If the results hold steady no matter what assumptions go in, that signals a genuinely strong channel. If the story changes based on the input, the signal is weaker than it may have appeared. Lorenzini contrasts this with relying solely on traditional error terms like MAPE, which can be overfit, describing his approach instead as actively testing the model rather than passively watching it.
In this video, you’ll learn:
How to identify when data signal is weak due to sparsity or lack of variance
Why traditional error terms like MAPE can be misleading or overfit
The methodology of perturbing model assumptions to test robustness
How comparing priors to posteriors reveals the true strength of a channel signal
Why proactive experimentation is more effective than passive model spectating
TIME STAMPFull Video Transcript: Testing Marketing Mix Model Robustness Explained
+–
LiftLab’s Jon Lorenzini explains how to test marketing mix model robustness by perturbing assumptions and comparing priors to posteriors.
[0:00 – 0:28] Transparency and Challenges in Data Modeling
With the Channel Signal Strength feature, what we are trying to do is provide a level of transparency that I do not think a lot of modelers do. Modelers want to provide maximum amount of confidence in the model so then they take it and they run with it and their product is well trusted. However, it is not always the case. If you have flat even spend, if you are not doing any variance or the variance is going with your demand, it is really hard to decomp and get those answers.
[0:29 – 1:02] Improving Model Confidence and Data Sparsity
There is no clear answer. We can get closer to it with expert opinions and pieces there, but what we want to do with our Channel Signal Strength is first surface that up. A little bit of how it works is if you have data sparsity, not live many days, you turn on only during tent pole events, that is hard to model. We also like to perturb the data; we say whatever we are putting in as our assumptions, let us change our assumptions and do they snap back to the same outcomes. That tells us we have a very strong signal, no matter what our opinions or the technical term is, priors posteriors.
[1:03 – 1:26] Prior and Posterior Analysis
We mess with the priors and then do the posteriors change, by how much, and if it is in the same direction or if it flips direction. Using that information, we can then start to say, hey, this is going to tell the same story regardless of what I put in as my prior, or what I put in is what I get out, in which case it is a lower quality signal.
[1:27 – 1:54] Programmatic Channel Strength Assessment
I think that this is a little bit better than your MAPEs or your other error terms, because those could be overfit. This is more of, I like to say, instead of spectating outside of the boxing ring, you are directly boxing with your data and your model to see if you push this side of the waterbed, does this side go up, and vice versa, and how does everything connect. That is what we did programmatically to be able to give you that channel strength score.
Key Lessons: Testing Marketing Mix Model Reliability
Surface underlying assumptions: Instead of assuming models are perfectly trusted, proactively surface the signal strength to reveal where data may be sparse or lack sufficient variance to draw accurate conclusions.
Prior and posterior analysis: Use the technical approach of messing with input priors to observe if resulting posteriors shift. If the story remains unchanged despite changing inputs, it indicates a high-quality, trustworthy signal.
Interactive modeling: Think of modeling like boxing with your data. By actively testing, such as observing how changes on one side of the waterbed affect the other, you can verify how channels connect rather than just observing outcomes passively.
