Measuring the true effect of your ads with Geo Testing

Figuring out an ad’s real effect is tricky. Clicks don’t tell the whole story. Even if the user clicked on the ad and made a purchase, we can’t be sure if the ad drove this sale, or if it would’ve happened anyway, especially if they saw multiple ads on different platforms. And sometimes, a user doesn’t click on the ad but makes a purchase because of it later. Attribution models can’t capture the true effect of the ad, so they give a biased image.

The solution lies in running tests to compare different scenarios: one where you advertise and one where you don’t. You want to create two different worlds which are similar in every parameter except for this specific ad you’re trying to investigate. Other campaigns can be going on at the same time too, as long as they are going on in both worlds. You don’t even care if the user interacted with the ad, because if the ad had any effect on the bottom line, a comparison between these two worlds will show it. You can also measure an entire platform, or any combination of ads, you just need to compare a scenario with it and a scenario without it. But how do you create and compare these worlds?

Beyond A/B Testing

A/B testing is probably the first thing that comes to mind when mentioning testing. You randomly split your audience into two: some see the ad, the others don’t. Then, you track the people in both groups and compare KPIs to see the real impact of your ad. Any difference between these groups can be attributed to the ads. But there are three big limitations:

• New privacy laws make it hard to track who sees your ads, and what they do later.
• If you’re advertising offline, tracking is almost impossible.
• Spillover effects. If someone who sees the ad tells their friend, and this friend then buys. This purchase is an outcome of your ad, but you’ll have no idea. This is a part of the effect of the ad you just can’t track with traditional methods.

So, we need a smarter approach.

The Solution: Geo Tests

Geo Tests is a smart workaround. Instead of focusing on individuals, you identify whole geographic areas, usually DMAs or stats. Some areas see the ads, and some don’t, and you compare KPIs in these areas. This could mean billboards in certain states but not in others. This approach dodges privacy issues since you’re comparing areas, not individual people, and can be easily tracked even for offline ads. But there’s a catch: you need to make sure the areas you’re comparing have similar enough trends in your KPIs prior to the test. Otherwise, you can’t really say the difference in the KPI during the test was because of the ads.

For example, if you place ads for cars in Iowa but not in New York, we can’t say the difference in sales in these states is only attributed to the ads. New York and Iowa are very different places, with New York having a denser population and greater reliance on public transportation, while Iowa has more rural areas and a higher dependency on personal vehicles. So you need to be smart when dividing areas into test and control, and even use weights to make sure these two pseudo-populations made from different areas have similar enough trends to be compared. If we do that, we can estimate the true causal effect of the ad, and calculate the lift, incrementality and all the usual metrics.

Keep In Mind

To get accurate results from Geo Tests, keep these in mind:

• Power Analysis: Should be conducted before the test begins and help you decide how big your test needs to be to be reliable.
• Trend: You want to make sure the two KPI trends of the geos selected have a similar pattern over time prior to the test. One way to check that is to investigate the difference between these trends, and see that it’s constant over time.
• Timing: Choose when to run your test carefully; timing matters. The effect of your ads changes when demand changes.
• Limitations: Geo Tests gives us a snapshot of a moment and a situation. We know how this ad works for the setting where it was tested. The situation includes other ads happening, promotions, what our competition did, ect. Usually, we recommend testing more than once, under different situations and times.
• These results can then feed into broader marketing strategies, like Marketing Mix Modeling (MMM), for a fuller picture of what works.
• Tools: Different tools are available to conduct Geo Tests, including some open source packages like Geolift by Meta. Most tools use a similar methodology, so you can choose what works for you!