Corrections, sequential looks and the choice of test decide what a number means. Posts on drawing a conclusion your stakeholders can act on with confidence.
In this post, we will dive into the various methods for addressing multiple comparisons, examining their strengths and weaknesses, and providing guidance on when each method is most appropriate to use.
The problem of multiple comparisons represents a fundamental clash between statistical inference and human intuition. In this blog post, we'll explain why our intuition leads us astray and review common methods used to address it.
In this blog, we’ll explore superiority tests and non-inferiority tests (also known as "do no harm" testing). We’ll examine their differences and provide a practical guide to designing and interpreting non-inferiority tests.
Peeking and early stop is one of the most common mistakes companies do across the board. Learn how sequential testing in the Frequentist approach allows it while maintaining control over false positive rates.
Choosing between one- and two-tailed hypothesis affects every stage of A/B testing. Learn why the hypothesis direction matters and explores the pros and cons of each approach.
Learn why deviations from normality are often not a significant concern in A/B testing, understand the pros and cons of using the t-test in such scenarios, and explore alternative methods to the traditional t-test.
Learn why Bayesian A/B testing can lead to misunderstandings, inflated false positive rates, introduce bias and complicate results rather than clarify them.
Corrections, sequential looks and the choice of test decide what a result means. We do this analysis on client experiments every week.
See how we work on a/b testing →