Geo-Based Experiments: A Full Walkthrough of Setting Up a Matched Market Test

We are going to talk today about running geo-based experiments, and not every marketer has run one already, and that was part of our motivation to simplify, standardize, and automate the process of running geo experiments. Another motivation was, these work well across every ad platform. We want to standardize and use the same recipe, and that is actually what is missing for marketers today. You don’t want to run one experiment design that is maybe designed by Facebook for that test, and a totally different methodology for Google. So, one size fits all by design. And where would we be running a test? We are going to run one today; our example is in Google Shopping, specifically in PMax.

The motivation for that could come from LiftLab. We may raise our hand and say, here is an area that you should consider testing, or it may come from our customer or their executive team: hey, this is a channel that we have always struggled to quantify, could you start over here? This is kind of the welcome screen of LiftLab, and today we are actually going to focus on one particular place, which is the Experiment Designer, so that we can show how to run a geo lift test.

When you go over to the Experiment Designer, your first question is, by default you will be running a Go Dark with Pacing test, and that is exactly what we are briefing today. The other alternative is a Spend Level test. Those tests are reserved for very large channels, where the capability or the appetite to go dark and not spend is not inside the comfort zone of the client, and so those are spend level tests, where you will spend down and spend up and spend BAU concurrently, and we will extrapolate out to what is the case where we don’t spend at all. Most examples we are going to go dark, and those are going to all be geo-based experiments.

As we go into this go dark test, we will have to give the experiment a name. We get to decide one or more business outcomes, and these business outcomes line up with data that was going to be observed. What happened in the areas where we are not spending? What happened anyway? Common ones we might see: people will analyze what happened to the site when we took away ads, what happened to the retail or offline sales when we took away ads. We might even want to zoom in on specific customer acquisition activity, maybe we are running a top-of-funnel media test and we want to know what happened to acquisition. And then, what are the product categories? Most commonly we are going to use the “all product” category, and we will go ahead and move along.

Now we are going to be asked, where in the media plan are we going to run this test? And I thought in this example maybe we would look at some of the shopping, and so we will look at Google Shopping, and inside of that channel we have multiple tactics, and so this test could be on some of the more traditional tactics. Maybe it would be more appropriate to say, what is going to be happening in our Performance Max? So let us go ahead and set this up as a Performance Max test.

And what we are going to be asked now is, what are the matched market pairs? What is the area of the country we are going to run this test? Now, we happen to have some of these pairs available in the account, and we can choose one for the test. We are going to go ahead and pick this one. This is treated versus control. We can see the area of the country where this is going to go live. We can also see it as a list, and these lists are downloadable. But let us go ahead and take this one, and we are seeing that the geos that we are using fall into these cells. We are not going to serve ads in about 16% of the country. So in this particular example, that is the holdout group, and that means we are going to continue to serve ads in a similar group to the control group, an outlier group where these are set aside, but we are going to keep serving ads as we would have, and some of the country that will not be analyzed. By default, we have a check to say, let us let LiftLab write these geos into the campaigns that we selected, over API. So we are going to let LiftLab do that.

The other part of Go Dark with Pacing is the pacing section, and here we need to choose one of the templates of what will the pacing look like. We are not going into detail on how the pacing in LiftLab works, but I will share today that the rule of thumb is, we are going to have an equal number of spend-up days as we are spend-down days. Right? So we are going to net out to the same budget. We can see this as a percentage or the raw spend, and we can edit this template if we need to. If there are certain days that were set to be spend-up and we want to swap them to spend BAU or spend-down, we can do that editing here.

And that kind of gets us close to bringing this home. We are going to be presented with a summary of all of the decisions that we made, and if we click on this “Schedule Experiment” button, many writes are going to happen. We are going to start writing into the Google Ads account all of the exclusions, and on a daily basis we are going to be evaluating, do we need to make changes to the pacing? That pacing could be changes in order, so that spend will be changed mildly, and the actual intervention might happen on the budget or on the trades, but it will net back out to the same budget. So, we are not going to schedule this experiment today, that will be for some other time, when we are ready to go live. This takes us through, in short order, all of the steps required, and shows the level of automation that has already been tackled on behalf of LiftLab customers who want to run automated experiments.