MAPE in marketing mix modeling measures overall prediction, but models are built for inference, understanding channel effects. This matters because a model can post a great MAPE while getting elasticities wrong, the number driving budget decisions. Without checking elasticities directly, marketers pile parameters onto weak data instead of fixing the real, low-quality inputs. This video explains how diminishing returns experiments engineer data so elasticities become reliable, not just prediction accuracy.
Why MAPE in Marketing Mix Modeling Measures the Wrong Thing
MAPE in marketing mix modeling measures the wrong thing, because agile marketing mix models exist to answer inference questions about individual channels, not to win a prediction contest. In this video, John Wallace, CEO of LiftLab, points out that the industry default, Mean Absolute Percent Error, only tells you how well a model predicted a validation period overall. It says nothing about how well any single channel was actually estimated.
Since these models are used for inference, Wallace argues equal weight belongs on marketing elasticities, the number that quantifies how a change in spend on a specific channel translates into a change in conversions. When an elasticity looks unreliable, his advice is blunt: don’t shoot the messenger. The problem is usually low-quality data going into that channel, not a modeling flaw that more parameters and corrections can paper over. The fix is to take ownership of the data itself by running diminishing returns experiments, which deliberately create variability and contrast that a flat spend pattern never would, giving the model genuinely better information to estimate that channel’s elasticity from.
In this video, you’ll learn:
Why MAPE is often an insufficient metric for marketing models
The crucial distinction between predictive modeling and statistical inference
How marketing elasticities quantify the effectiveness of channel spend
The dangers of relying on low-quality data for business decisions
How to use experiments to engineer better data for more accurate results
TIME STAMPFull Video Transcript: MAPE in Marketing Mix Modeling Explained
+–
LiftLab CEO John Wallace explains why MAPE is the wrong metric to chase in marketing mix modeling, and how experiments fix unreliable channel elasticities.
[0:00 – 0:33] Predictive Modeling vs. Inference
By definition, when we’re building marketing mix models, we’re building predictive models. And like many predictive models, our models can be used for prediction and other uses for inference. What I find is that quite often there’s too much emphasis placed on the predictive quality of the model, which, in the industry, is quite often characterized or quantified as what’s called the MAPE, and that’s a fancy acronym for Mean Absolute Percent Error. What that’s telling us is how good of a job, if we’re looking at validation, the model is doing as we enter that validation period.
[0:33 – 0:52] The Role of Marketing Elasticities
And since we’re using these models for inference, I think that we should be equally focused on how well we think we’ve estimated every one of our individual marketing channels. And the number that we’re using to quantify the effect of a channel is what we’re building, are marketing elasticities. Just how good are those marketing elasticities at representing the impact of changes in spend on changes in conversion?
[0:52 – 1:11] Addressing Unreliable Data
If we find the condition where one of our marketing elasticities doesn’t appear reliable, I like to sometimes tell clients, “Don’t shoot the messenger.” We may have low quality data coming in for that marketing channel, and we could try to load that model up with lots of fancy parameters and corrections and adjustments, but it’s still going to be low quality data.
[1:11 – 1:30] Engineering Better Data with Experiments
What we like to do is turn our attention to how can we take ownership of this, and how can we engineer better data. And that’s what we do when we run experiments. These diminishing returns experiments are designed to create the kind of variability and contrast in the data that will help us quantify the marketing elasticity for that channel, and take that new information and plug it into our model.
Key Lessons: Why Elasticities Matter More Than MAPE in Marketing Mix Modeling
Modeling limitations: Predictive quality measured by MAPE does not guarantee reliable inference for individual marketing channels.
Performance metrics: Marketing elasticities are essential for measuring the direct impact of spend changes on conversion outcomes.
Data integrity: Attempting to fix poor data with complex parameters is less effective than addressing the root quality issues.
Experimental improvement: Running diminishing returns experiments creates necessary data contrast, allowing models to learn and self-heal.
