Experimental Design Meets Econometrics: The Idea Behind Modern Marketing Measurement

I started LiftLab because I am a measurement person. I have been doing measurement all of my career. I have measured some pretty diverse things; I looked at half of the credit card transactions in the world running on the Visa Network. We have predicted recalls for cars that were going to need to come into the garage, and then focused more and more on marketing with some fairly large, healthy brands, Williams Sonoma, Gap, Apple, companies like that. I made a discovery several years ago around the area of multi-touch attribution, and that was a really hot practice. It was to look at the path to purchase, the customer journey, and figure out what really drove that sale.

The algorithms that we put together for that were super hot. It was borrowing from the areas of biostatistics. Instead of exposing people to different levels of exercise and different levels of medicine or different combinations of drugs, it was really about how many clicks did they have and how many exposures did they have. Would that go on instead of them dying, it was, were they buying. It was a lot more fun problem, actually. So, adapting those techniques gave us a really strong algorithm. That company was ultimately acquired by a team at MarketShare, and MarketShare was acquired by Neustar. The multi-touch attribution company was called DataSong.

My realization was that the data that we were using for multi-touch attribution in that era was going to go away, and it was going to go away due to, in my opinion, overdue changes in privacy. The data that we were seeing, although it was anonymous, was not going to persist. That premonition was actually ultimately proven right. I remember one of our customers right now, from Thrive, his name is Devon, he used to be on the marketing team at Sephora, and he came jumping onto a weekly status call saying, “What are we going to do about iOS 14.5? It’s going to destroy all the data that we’re using for measurement. What is LiftLab going to do to react to that?” I got to smile a little bit and say, “We’re going to do nothing. Absolutely nothing.” Everything that we’ve designed from the ground up at LiftLab was towards the direction of, how do we work with data that’s privacy-compliant. And the pillars of LiftLab to do that, so these are all new data sets, they’re all new algorithms. This is not kind of a redo of my former company, of DataSong; it’s a fresh start.

It’s been four years in the making, and it started with, let’s embrace the ability to set up marketers and empower marketers to make data. They need data for measurement, and they need to make it. And what we mean by “make data” is, can you run experiments? Every marketer loves a good experiment. We don’t really have anyone who says experiments are a bad idea. But the practices of running experiments, that’s where there was room for improvement. Traditionally, experiment frameworks that we’re using in marketing were borrowed from other fields. They actually come out of agriculture if you really look back at the literature, maybe some of it’s been done in industrial settings, and it’s called an A/B test. Like I said, every marketer loves themselves an A/B test. And you saw this a lot in marketing, on the websites, website performance.

There are companies like Optimizely who set you up to do a plethora of A/B tests. It turns out that’s not a great strategy for paid media, and we recognized that early on. A/B tests are sometimes, in paid media, referred to as incrementality. The dirty secret of those tests is so many of them are run and they’re not statistically significant. That’s kind of the bad news, or the spoiler. And we wanted to do something that was stronger. We wanted to set marketers up to run tests that work out of the box a lot more often. I proposed an idea to one of my dear advisors, his name is Dr. Hanssens. He’s out of UCLA; I would call him the godfather of the marketing mix model. And I approached Dr. Hanssens and I said, “I have a question for you. I have this idea to combine experimental design with econometrics. Can you tell me if this is a good idea, or you want to talk me out of it?” And he smiled and said, “I think you’re on to something.” And that was the nexus for LiftLab.

Can we start to make data that is privacy-compliant? And can we analyze it with a stronger instrument? Right, the A/B test is too brute force, too blunt of an object. We need something, we need more, not a magnifying glass, we need a microscope. And that’s what’s happening at LiftLab. All the experiments that are running there, all the brands are kicking off experiments; they follow a very strong sampling design. And we have more power, we will call it in statistics, right? We have more statistical significance. And it required us rethinking it and saying, an A/B test gives us a point in time, and it gives us a point estimate. What’s the incrementality?

In our field, we’ll hear things like, “the incremental ROAS was a 3.” But what that doesn’t tell you is, and if I increase my spend, what’s the incremental ROAS? Because the answer is no longer valid. And trying to run another test and another test to figure that out takes too long. And if along the way some of those tests aren’t significant, the story falls apart. I mentioned the term econometrics, what we also hear referred to sometimes. We want to look at this as longitudinal data. And so we’re collecting data sets, our customers are collecting data sets, to give them a pile of evidence about, how effective is this marketing?

The feedback from those earlier customers was, “these experiments are hot, but they don’t tell me what to do on every single channel every Monday morning.” And we went back to our modeling roots, this time not using biostatistics but using econometrics, or what’s called a marketing mix model. And we wanted to bring some fresh thinking to marketing mix models as well. So we kind of revolutionized the experiment world for paid media. And on the mix modeling side, the number one frustration that people had was, “I get a mix model result three, four, five months after the year ended.” It takes way too long. And as much of an appetite as there was to consume the information out of the mix model, it was deemed somewhat irrelevant. And so we now have a mix model that’s published every Monday morning. These are signals that are clear auxiliaries and replacements over time for working with last-click data that they would typically use on a Monday morning. They are calibrated by the experiments. So that’s another revolution. How do I address that some of the data in my mix model is low-signal data, and no matter what I do to that data, I’m not going to increase its fidelity? I need a means to make data, and that comes back to the experiment side of LiftLab. And the other thing that we did in our mix model design was to build it in a hierarchy, like a marketer would think about. So I have paid social, but I have that in different platforms. And within paid social, I have different parts of the funnel. Those all need to be able to be broken out, upper funnel, mid funnel, lower funnel. And inside that funnel, I might have one or I might have 10 campaigns. So it doesn’t suffice for me to know, “hey, my lower funnel is working well.” I need to know which of the campaigns are working well. So that drill-down capability, that granularity, is also enshrined in LiftLab.