Bell's Blog

When business constraints limit A/B testing treatment exposure, analysts must choose between the statistical integrity of a balanced design (e.g., 10/10) and the speed of an unbalanced design (e.g., 10/90). The unbalanced 10/90 split, which uses full traffic to dramatically accelerate the experiment, comes with risks like inflated false positives from skewed metrics, Sample Ratio Mismatch, and resource contention bias. Read on as we examine this critical trade-off to help you balance speed with statistical validity.

Our “Geo Testing: Unlocking True Incrementality” webinar explored how teams can measure real-world impact when A/B testing isn’t possible - using geographic experiments and synthetic controls to reveal true incremental lift in marketing and product initiatives.

Too many teams stop after losing one A/B test - but that’s where real optimization begins. This blog explores how structured iteration, smart testing tactics, and small, data-driven tweaks can turn failed experiments into lasting conversion wins.

In this post, we will dive into the various methods for addressing multiple comparisons, examining their strengths and weaknesses, and providing guidance on when each method is most appropriate to use.

The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.

Randomization is one of the cornerstones of AB testing. But why is randomization so important, and how can we achieve it? In this post, we'll address these two key questions. Because without proper randomization, no A/B test can be considered valid.

This guide aims to give you practical principles to develop A/B testing mindset. Principles that will help you survive, thrive, and systematically reach the 20-30% test success rate that creates dramatic impact in your bottom line.

In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.

In this blog, we’ll explain why running multiple experiments simultaneously is not only feasible but also beneficial, explore its key advantages and potential challenges, and share best practices for successful implementation.

A/B testing analysts often encounter skewed distributions for different KPIs. Learn why outliers can be problematic, explore the challenges of identifying and handling them, and get practical guidelines for dealing with them effectively.

Our "Sweet Test o' Mine" webinar debunked some of the most common A/B testing myths, highlighting the importance of testing often, iterating quickly, and trusting data over intuition.

Do you check for Sample Ratio Mismatch (SRM) in every test? Learn why checking for SRM is crucial, explore common reasons it occurs, and learn how to detect, diagnose, and address it.

Many analysts make fundamental mistakes that can misrepresent or undermine the test's findings. This blog highlights common reporting errors and offers practical guidance for presenting tests results effectively.

A common issue in A/B testing is when the allocation point does not align with the exposure point. Learn why it matters, how it impacts your a/b tests, and how to address it.

Peeking and early stop is one of the most common mistakes companies do across the board. Learn how sequential testing in the Frequentist approach allows it while maintaining control over false positive rates.

Choosing between one- and two-tailed hypothesis affects every stage of A/B testing. Learn why the hypothesis direction matters and explores the pros and cons of each approach.

Learn why deviations from normality are often not a significant concern in A/B testing, understand the pros and cons of using the t-test in such scenarios, and explore alternative methods to the traditional t-test.

Learn from top expert, Ronny Kohavi, how to build infrastructure required for A/B testing, foster a culture that encourages experimentation, and ensure your teams learn valuable lessons from failures.

Understanding why the uplift seen in A/B tests often differs from real-world outcomes is essential. Explore key reasons behind this discrepancy and insights on how to manage expectations effectively.

CUPED is a powerful technique that not only enhances the sensitivity of your tests but also helps in resource optimization by reducing the number of subjects needed for experiments.

When your advertising efforts are up and sales are increasing, that’s great news, but it doesn't necessarily mean the ads are the reason. Here are three popular tools to measure the lift of a campaign.

Ever wondered why your A/B tests aren’t delivering the results you expected? Reveal the most common A/B testing mistakes made by top companies and how you can avoid them.

Three key trends shaping the future of MMM: the emphasis on causality, the adoption of Bayesian methods, and the push towards real-time analysis.

Robyn, Meridian, Orbit, or custom-made MMM? In this blog post, we compare leading free, paid or custom MMM tools, to help you decide which best fits your needs.

Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.

In today’s privacy-focused era, the different attribution models create many blind spots for marketing analysts and decision makers. However MMM & Geo Tests can help.

Figuring out an ad's real effect is tricky. Clicks don't tell the whole story and attribution models fall short. The Solution: Geo Testing.

How to plan your marketing spend while measuring the true effect of any marketing activity on your bottom line? The answer lies in Marketing Mix Modeling (MMM).

Traditional methods often fall short in measuring the true impact of marketing strategies. Here's how Geo Tests can resolve common marketing challenges.

In A/B testing, you often have to balance statistical power and how long the test takes. Learn how Allocation, Effect Size, CUPED & Binarization can help you.

Measuring complex user journeys is nearly an impossible task without the proper tools. Learn how MMM & Geo Tests can help advertisers measure the impact of their cross-marketing efforts.