Incrementality testing for paid media replaces the go-dark A/B test, cutting spend and waiting to see sales fall. This matters because sales rarely fall enough to be detectable, and go-dark tests assume marketing runs identically every day. Without a better design, marketers misread an effect as noise or never learn how far a channel could scale. This video explains how diminishing returns experiments with spend pacing replace traditional A/B testing.
Why Incrementality Testing for Paid Media Beats the Traditional Go-Dark A/B Test
Incrementality testing for paid media beats the traditional go-dark A/B test because a go-dark test asks sales to drop by enough to be statistically detectable, and in most paid media, they simply don’t drop that much. In this video, John Wallace, CEO of LiftLab, admits he loves A/B testing as much as any marketer, but years of running them exposed the cracks: randomization rarely produces two truly identical groups, and the standard fix, large sample sizes, only partially compensates. In paid media specifically, the go-dark test’s core method, pull spend and watch sales plummet, doesn’t hold up, since sales typically dip only slightly, making the effect hard to distinguish from noise.
Wallace explains that LiftLab set out to design something better: diminishing returns experiments that still go dark but add pacing to the remaining spend. That directly corrects a hidden assumption baked into every A/B test, that marketing runs identically every day, which is rarely true and, in Wallace’s view, shouldn’t be. Deliberately introducing contrast into spend and analyzing the result as a longitudinal data set, rather than a single before-and-after snapshot, doesn’t just prove whether marketing works. It answers the harder, more useful question of how far that spend can actually scale.
In this video, you’ll learn:
Why traditional A/B test assumptions often fail in practice
The inherent flaws found in common go-dark paid media experiments
How diminishing returns experiments offer a more robust analytical alternative
The importance of considering marketing timing and spend pacing in analysis
How to shift from testing to scaling marketing investment effectively
TIME STAMPFull Video Transcript: Incrementality Testing for Paid Media Explained
+–
In this transcript, LiftLab CEO John Wallace explains why traditional A/B testing falls short in paid media, and how diminishing returns experiments with spend pacing fix it.
[0:00 – 0:26] The Limitations of A/B Testing
I challenge you to show me a marketer that doesn’t want to run experiments. It’s in our nature, and almost by default, when you say the word experiment, it’s completely associated with A/B testing. I love A/B testing just like every other marketer, but having done enough of them in my career, I realize that sometimes they don’t get us all the way to the outcome that we need. It turns out that A/B tests are really intuitive; that’s what we like about them. But to have that simplicity, they make a lot of assumptions.
[0:26 – 1:12] The Flaws in Paid Media Go-Dark Tests
They’re assuming that, due to randomization, these two groups are going to be exactly identical, and that’s almost never true. We try to mitigate that by having power tests and sample sizes that are large enough to kind of overcome that. In paid media, when we do an A/B test, the most common one is a go-dark test. We just take away media and then we go and observe and say, “How much did sales plummet?” The problem is that sales don’t plummet; they go down just a small amount, and the smaller that amount is, the harder it is to detect that that was truly caused by the absence of media. For me, this really bothered me, and I felt like the failure rate of go-dark tests was just too high. I felt like we didn’t want to perpetuate that and enshrine it in software.
[1:13 – 1:56] Transitioning to Diminishing Returns Experiments
So when we got started at LiftLab, we went out of our way to think about what could be new experiment designs. How can we somewhat retire the A/B test? What we turn to are diminishing returns experiments. The idea is to go dark and to put pacing on the rest of the spend. What that’s addressing is one of those assumptions of the A/B test; we want to correct it. One of the assumptions is that the marketing happened, it just happened, and those tests ignore the timing of the marketing. They essentially assume that the marketing is equal every day, which is almost never true, or at least I recommend for you to not have it be true. But we want to go a step further and say, “Let’s exacerbate that.” Let’s put some new forms of contrast on the marketing spend and pull that into our analysis.
[1:57 – 2:21] Treating Data as Longitudinal Sets
Really what’s happening here is we’re graduating from A/B tests to treating this data as experimental data sets that are longitudinal. And once you have opened the door to that, you have a lot more success at understanding whether marketing is effective. But you’re setting yourself up to answer a much harder question, which is, “Now that I know the incrementality, how far can I scale this spend?” I can guarantee you that once you have a taste of it, you’ll never want to go back.
Key Lessons: Moving From A/B Testing to Diminishing Returns Experiments
Experimental limitations: Conventional A/B tests rely on randomization assumptions that are rarely met, leading to inaccurate conclusions.
Measurement challenges: The common go-dark method often lacks the sensitivity required to attribute sales changes directly to marketing absence.
Strategic shift: Transitioning to longitudinal experimental sets allows for a more nuanced understanding of marketing performance over time.
Growth optimization: Analyzing spend contrast helps businesses transition from simply proving effectiveness to identifying optimal scaling limits.
