Six Tips for Running Geo Experiments the Right Way

It is often said that experience is the best teacher, and geo-testing is no exception. Running high-quality geo-experiments involves a learning curve, where performance improves with practice and accumulated knowledge. In this blog, we crystallize our experience from conducting dozens of geo-tests across the industry into six practical tips. This is essential reading for anyone who not only wants to run geo-experiments, but to run them the right way.

Allon Korem

Allon Korem

Chief Executive Officer

8 min read

Six Tips for Running Geo Experiments the Right Way

It is often said that experience is the best teacher, and geo-testing is no exception. Running high-quality geo-experiments involves a learning curve, where performance improves with practice and accumulated knowledge.

In this blog, we crystallize our experience from conducting dozens of geo-tests across the industry into six practical tips. This is essential reading for anyone who not only wants to run geo-experiments, but to run them the right way.

Tip #1: Tie Every Test to a Business Decision

A geo-test is not just an analytical method; it is a decision-making tool. If you do not define upfront what question the test is meant to answer and how the results will translate into concrete actions, you risk ending up with a set of insights but no real business impact.

Geo-testing is typically used in three main scenarios: evaluating a new marketing channel, validating the incremental impact of an existing one, or optimizing budget allocation. Each test should therefore be tied to a clear business decision, whether to scale a campaign, reduce investment, reallocate budget, or even shut down an initiative.

Acting on data-driven insights may seem straightforward, but in practice, it is often more difficult. Organizations are usually quick to act on positive findings, such as evidence of strong incremental impact. The real challenge comes when results contradict existing assumptions, for example, showing that a long-running channel generates less incremental value than expected. In these cases, maintaining the status quo can be easier than reallocating budgets or discontinuing established initiatives. As a result, some of the most valuable insights from geo-testing may be acknowledged but ultimately left unused.

How does this work in practice? Take a profitable company that sets a minimum incremental ROAS target so that every marketing dollar meets its profitability requirements. The geo-test can then be built around a simple decision rule: channels that clear the threshold are candidates for more budget, while those that fall short are candidates for cuts or reallocation. This means a channel can drive real lift and still fall short if its return isn’t high enough to meet the company’s goals.

Tip #2: Choose the Right Geographical and Temporal Granularity

The shift from user-level to geographical-level experimentation requires analysts to make two fundamental design choices: which geographical units to use (e.g., states or DMAs; figure 1) and at what temporal granularity to analyze them (e.g., daily or weekly).

Before diving into practical recommendations, it is important to understand why these choices matter. In user-level experiments, it is well known that increasing sample size improves statistical power (the ability to detect an effect when one truly exists). The same principle applies in geo-testing: increasing the number of regions and the number of observations per region over time generally increases power.

However, unlike classical experimentation, this is not a simple case of “more is better.” In geo-experiments, increasing the number of units introduces additional risks. In the geographical dimension, finer segmentation can increase susceptibility to interference between regions (spillovers) as well as measurement contamination due to imprecise location signals and user mobility. In the temporal dimension, using finer units can increase variability and reduce signal stability.

Figure 1. Example of different levels of geographical granularity in the United States. States (left) provide fewer geographical units, with a lower risk of spillover effects. Designated Market Areas (DMAs; right) provide more units with relatively similar media and marketing behavior, but may be more susceptible to spillover between neighboring areas.

As a result, selecting the right granularity becomes an exercise in balance: the design must be sensitive enough to detect meaningful effects, while remaining coarse enough to preserve experimental integrity. In practice, we often recommend DMAs (where available) for geographic units and daily aggregation for temporal analysis, providing a good balance between statistical power, noise, and risks such as spillovers and leakage.

Tip #3: Ensure technical feasibility of the test

Decisions about experimental granularity are important, but they are only meaningful if they can be implemented within the company’s technical constraints. In other words, a well-designed geo-test on paper is useless if it cannot be executed in practice.

For example, if the experimental design is based on DMAs as the unit of randomization, it is essential to ensure that the organization’s advertising systems can actually allocate campaigns at the DMA level rather than only at broader geographic levels such as states. Similarly, if the temporal granularity is set to daily analysis, the data infrastructure must support reliable daily-level tracking and aggregation, rather than being limited to weekly reporting cycles.

Ultimately, a sound geo-test must be both statistically valid and technically feasible.

Tip #4: Use Residuals to Validate Your Design.

You’ve planned your geo-test and defined a split between control and treatment regions. Before the test goes live, it is critical to validate that the two groups are well matched using pre-test data.

A common mistake is to rely solely on aggregate pre-test comparisons between the two groups. While these may appear reassuring, they can mask important structural differences. A more reliable approach is to examine the distribution of residuals, the differences between control and treatment regions over the pre-test period.

In a well-balanced design, the residuals should exhibit three key properties:

  • Bias: Is one group consistently higher than the other? Residuals should average close to zero. A persistent positive or negative difference suggests systematic imbalance between the groups.
  • Drift: Does the difference change over time? The residuals should remain relatively stable throughout the pre-test period. A systematic increase or decrease may indicate that the groups are becoming less comparable.
  • Autocorrelation: Are today’s differences related to yesterday’s? Residuals should fluctuate largely randomly over time. If a positive difference tends to be followed by another positive difference (and similarly for negative differences), it suggests a persistent pattern rather than random noise, which may point to a structural difference between the groups.

Figure 2. Residual Patterns in Pre-Test Diagnostic Design. Comparison of an ideal well-balanced design (top-left, centered at zero with random variation) against three common issues: bias (constant offset), drift (time trend), and autocorrelation (non-random clustering).

Together, these diagnostics provide a robust validation of the experimental design and increase confidence that post-test differences reflect the intervention rather than pre-existing imbalance.

Tip #5: Run Early Checks When the Test Goes Live

Few things are more frustrating than investing weeks in a geo-test only to discover that a setup issue invalidated the experiment from the start. This is not a hypothetical risk; it is a scenario we have observed across multiple organizations.

To avoid this, treat the first days of the test as a critical validation window. Verify that:

  • Ad delivery follows the experimental assignment.
  • Budgets and spend match the planned allocation.
  • Control regions remain isolated from unintended exposure.
  • Performance metrics show no extreme outliers or unexpected spikes that could signal tracking issues, configuration errors, or leakage.

Figure 3. Early warning patterns in the first days of a geo-test. Daily spend (top) and conversions (bottom) for treatment and control. Left: a valid setup, where only treatment receives spend after launch, producing a clear lift. Middle: a missing region, where a geo-classification error makes treatment data drop to zero for several days. Right: leakage, where control regions receive ads and the measured lift shrinks.

When marketing investments reach millions of dollars, the cost of detecting issues late is simply too high. Early quality checks protect both the budget and the validity of the experiment, ensuring that results reflect the intervention rather than technical artifacts.

Tip #6: Get to Know Your Marketing Channel

A well-designed geo-test requires more than good data; it requires understanding how each marketing channel behaves and how its dynamics can affect experimental results.

Invest time in learning the platform’s learning phases, response patterns, and timing of impact, and consult channel experts when needed. To illustrate how channel knowledge can inform geo-test design, consider the following examples:

  • Algorithmic channels may have an initial learning phase during which performance is unstable as the system optimizes toward users who are more likely to convert. Early performance may therefore not reflect steady-state impact, making it useful to exclude this period from the analysis to avoid bias from transient effects.
  • Audio or radio may have delayed effects, as users often require repeated exposure before responding. The design should therefore account for lagged responses and longer observation windows.

Ultimately, each channel has its own response curve and timeline. Aligning the geo-test design with these dynamics is essential to avoid misinterpreting results or missing true effects.

One last tip

Perhaps the most important point is this: even if geo-testing feels more complex, don’t abandon it when user-level randomization isn’t feasible. Measuring marketing impact is too important to rely on methods that capture correlation rather than causation. Our goal in this blog was to equip you with the foundational knowledge needed to apply this approach.

The takeaway is simple: invest in geo-testing expertise; it is the most reliable way to estimate whether campaigns truly drive incremental impact. Without it, you risk making high-stakes decisions on evidence that reflects correlation rather than true incremental impact.

Share this postLinkedIn

Geo Testing at Bell Statistics

We plan, run and analyse geo experiments to measure what your marketing actually caused.

See how we work on geo testing →

More on geo testing

All posts →