I want it that way: Building experimentation infrastructure & culture

In the latest A/B Talks by Statsig, I hosted Ronny Kohavi for a talk about the fundamentals of effective A/B testing in mid-to-large organizations. We discussed key topics such as the infrastructure required for A/B testing, fostering a culture that encourages experimentation, and ensuring we learn valuable lessons from failures. Ronny is a highly respected thought leader, a seasoned expert in experimentation, and a former executive at Microsoft, Amazon, and Airbnb.

Key takeaways:

Organizations can be at various stages of maturity when it comes to their experimentation practices. It’s crucial to recognize organizations' current stage and focus on steady progress, rather than trying to reach the highest level right away.

Organizations must accept that failure is an inherent part of the experimentation process. Success rates are typically low (between 12%-33%), and when factoring in the false-positive rates, they drop even further. A good approach is to roll out features on a smaller scale, for instance on a single platform (desktop, Android, or iOS), and then expand based on positive results.

While useful for specific purposes like code refactoring, defensive testing (non-inferiority testing, which ensures no metric is negatively impacted beyond a set threshold) isn’t generally recommended for growth-driven experimentation.

Organizations should leverage experimentation tools to complement their existing processes. Implementing effective experimentation requires a significant investment, and having the right tools can make the process smoother and lead to better outcomes.