The MMM cold start problem happens because a new channel has no historical data, so a model can’t say anything reliable. This matters because waiting months delays answers, and low-signal data separately lets models bend toward intuition instead of truth. Without a faster source of evidence, teams wait too long on channels or let opinion steer established ones. This video explains how geo-experimentation solves both problems in weeks, not months.
Why the MMM Cold Start Problem Shares a Root Cause With Model Manipulation
The MMM cold start problem shares a root cause with model manipulation, because both come down to the same thing: not enough real evidence, whether that’s a new channel with no history or an established one running on low-signal data. In this video, John Wallace, CEO of LiftLab, frames these as the two “dirty secrets” of marketing mix modeling that come up constantly with enterprise marketing and data science teams.
The first secret is that a marketing mix model can be made to say almost anything. Wallace recalls being told a channel’s ROAS was “too high” and simply adjusted lower to match someone’s intuition, a symptom of low-signal data producing wildly different answers across iterations, leaving the people building the model, in-house or outsourced, essentially filling the gap with opinion. LiftLab’s answer is to expose exactly where that low-signal data is entering the model and replace it with real evidence from LiftLab’s incrementality testing engine run in weeks rather than the year it might take to naturally accumulate enough history.
That same capability solves the second secret, the cold start problem for new channels. Rather than waiting months for a channel to build up its own history before the next model refresh, Wallace recommends getting the channel live, satisfied with its in-platform metrics, and then immediately running an incrementality experiment on it, delivering a real answer and a dashboard update far sooner than the traditional wait would allow.
In this video, you’ll learn:
Why traditional marketing mix models can be easily manipulated
How low-quality data forces an over-reliance on subjective opinions
The challenges of the cold start problem for new marketing channels
How to use geo-experimentation as an alternative to improve model signal
Why experimental data provides faster, more accurate insights than waiting for long-term historical data
TIME STAMPFull Video Transcript: Incremental Revenue Measurement Explained
+–
LiftLab CEO John Wallace shares two dirty secrets of marketing mix modeling, models that can say anything and the cold start problem, and how geo-experimentation fixes both.
[0:00 – 0:18] Introduction: The Dirty Secrets of MMM
I call this conversation the dirty secrets of MMM, and this is a conversation that’s happening with the marketing executive or data science team of a large enterprise, and the two things that we focus on are that you can make marketing mix models say anything you want, and they are, like any model, exposed to the cold start problem.
[0:18 – 1:02] Secret 1: MMMs Can Say Anything
Let’s drill into those. I can characterize it how the marketers say it. I was presented a marketing mix model one time, and I said that the ROAS for one of these channels was too high, it didn’t pass my intuition, and they said, that’s okay, we’ll make it lower. That’s a good example of it. When you’re building a model and going through many iterations, and you have low signal data, it may show you one time that the ROAS is really low, and another time that the ROAS is really high, when the truth is somewhere in between. What this means is that the people that are building the model, whether it’s in-house or the firm that you hired, you’re relying a lot on their opinion when the data is low quality.
[1:02 – 1:25] The Alternative: Geo-Experimentation
Our alternative to that is to shine a light and show you where the low signal data is coming into this model, and let’s use the LiftLab geo experimentation platform to go collect better data. We use experiments and experimental data as an antidote to low quality data, and it gives us a way to collect data thoughtfully, and not try to recollect data for a year, but get that experiment to run in a few weeks and inject that new finding, that new evidence, into our model.
[1:25 – 1:53] Secret 2: The Cold Start Problem
The same feature solves the second problem, which is the cold start problem. We highly recommend to customers, when they’re going to work with a new channel, go get the new channel up and running, put some money through it, get the creative work, and get satisfied with the in-platform metrics, fall in love with those metrics, and then go and get the incrementality. This is a fresh alternative to, let’s go collect data for several months and put it in the next model build that’s around the corner. They get these results much faster, and they’re reflected in their dashboard much sooner.
Key Lessons: Solving the MMM Cold Start Problem and Model Manipulation
Model integrity: Relying on low-signal data introduces bias, often leading firms to adjust models until they align with intuition rather than truth.
Data quality: Implementing a geo-experimentation platform helps shine a light on areas where poor data enters the model, acting as an antidote to inaccuracy.
Efficiency: Instead of waiting months or years to collect data, running short-term experiments allows teams to inject fresh evidence into models quickly.
Channel scaling: For new marketing channels, the best practice is to optimize for platform metrics first before verifying true incrementality through experimentation.
