Marketing experiment results are useful no matter what they show, since confirming or correcting a model both help. This matters because low-confidence data is exactly when an experiment replaces a shaky assumption with real evidence. Without that willingness to revise, marketers keep overfunding or underfunding channels, losing profit either way. This video explains why pulling back or scaling up are both wins once an experiment shows which is correct.
Why Marketing Experiment Results Are a Win No Matter What They Show
Marketing experiment results are a win no matter what they show, because an experiment that confirms a model saves you from second-guessing it, and one that contradicts it just found money you were about to leave on the table or waste. In this video, John Wallace, CEO of LiftLab, describes running experiments on channels LiftLab has already modeled, something he says is fairly unique in the industry, precisely to put the model’s own results to the test.
Customer reactions split in an interesting way: some find it reassuring when the experiment simply confirms what the model already said, while others see a confirmation as a missed chance to learn something new. Models are most likely to get revised when the underlying data going in had low confidence to begin with, which is often exactly the motivation for running the experiment in the first place. Wallace frames both possible outcomes as genuinely good news: pulling back spend on an ad platform means the economics didn’t hold up and you get to reclaim that money as profit, while finding room to spend more means you were leaving money on the table and get to capture it. Either way, the data does the talking, and clients are coached ahead of time to be ready to act on whatever it says.
In this video, you’ll learn:
How to use experiments to validate marketing mix models
Why low-confidence data motivates the need for further testing
The benefits of questioning model assumptions to drive accuracy
How to interpret experimental results to optimize ad spend
The importance of being prepared to act on data-driven findings
TIME STAMPFull Video Transcript: Interpreting Marketing Experiment Results Explained
+–
In this video, LiftLab CEO John Wallace explains why every marketing experiment result, whether it confirms or contradicts the model, is worth acting on.
[0:00 – 0:21] The Role of Experiments in Model Validation
I look forward to discussing with customers and anticipating the reaction when we’re going to run an experiment on a channel that we’ve already measured in a model. That’s something that we do a lot of at LiftLab. It’s fairly unique to us that we’re putting under the lens, or putting into question, the results of our model using experiments, and in some cases, they’re going to come awfully close.
[0:21 – 0:48] Addressing Low-Confidence Data
One reaction when we get that result from a customer might be, “Well, that’s really reassuring that your experiment ran and it’s confirming what the model was saying all along.” Some people feel disappointed; they look at it as an opportunity to really revise what was in the model. The models are more likely to get revised when we have low confidence in the data going into the model in the first place, and that’s often our motivation for running an experiment, is to say, “Here’s a corner of the model that needs some attention, needs some love,” and we’re going to use an experiment to go gather better evidence.
[0:48 – 1:40] Acting on Experimental Findings
When we’re setting out to do that, and the clients are running these experiments in their ad accounts, I love to watch their reaction when we say that, you know, we might possibly have to revise down the economics of that ad platform. Like, we’ve got a better set of data now, and it’s saying that we should pull back spend. That’s actually a win-win opportunity, whether you get the answer to pull back spend or increase spend. Because when we’re pulling back spend, we’re just saying we’re going to spend a little bit less, and we’re going to increase our profit by doing that, right? So it’s an opportunity to reduce waste. Of course, the complement’s really fun too, when you have a chance to spend up; if you’re leaving money on the table, you’re going to increase profit by spending more. So we’re prepared for any of those outcomes; we talked to customers about this ahead of time, reassure them that whatever we get, it’s the evidence, it’s the data doing the talking, and be prepared to action whatever outcome we find.
Key Lessons: Turning Any Marketing Experiment Result Into a Win
Model validation: Using experiments helps confirm if your current marketing models are accurate or require adjustments based on real-world evidence.
Data confidence: When you have low confidence in your input data, running controlled experiments provides the necessary clarity to strengthen your overall model.
Strategic agility: Be prepared to pivot your approach, whether that means pulling back on unprofitable channels or increasing investment where you are leaving money on the table.
Efficiency gains: Every outcome, whether it involves cutting waste or scaling up, serves as a win-win opportunity to enhance your marketing profitability.
