Sweet Test O' Mine: Rocking the truth in experimentation

Experimentation is a powerful tool, but it’s easy to get lost in the details, overcomplicate things, or fall for common misconceptions. In our latest webinar, "Sweet Test O' Mine: Rocking the Truth in Experimentation", Tim Chan and I tackled some of the biggest myths and challenges in A/B testing, sharing hard-earned insights from years of experience helping companies optimize their experimentation programs.

Key takeaways:

Stop worrying (too much) about interaction effects - Running concurrent experiments is usually better than slowing down testing velocity. Most tests don’t move the needle much, and research (hello, Microsoft!) suggests interaction effects are rare.

Yes, you should test everything - Even the tiniest changes can have unexpected consequences (like a simple bug fix revealing a hidden issue). If experimentation is frictionless, there’s no excuse not to test.

You don’t need a million users - Small companies can experiment too! The key is testing changes that have big effects rather than micro-optimizations.

Holdouts are powerful (but complex) - Long-term holdouts help measure cumulative impact, but they require leadership buy-in and significant engineering effort. Use them when necessary - not just because you can.

Significance results "don't care" what is the MDE - The minimum detectable effect (MDE) is for planning. If results are statistically significant, they are significant, even if the observed lift is below the MDE. Don’t mix up planning and analysis!