How many changes should you A/B test?

Optimization (CRO) teams are often required to balance the earning and learning potential of changes. This post explores some strategies that teams can use.

trade-off
Source: MIT Sloan Management Review

You test one change at a time.

sample sizes and test durations for different levels of lift
sample sizes and test durations for different levels of lift
Sample sizes and test durations for different levels of lift assuming one-tailed test, 80% power and 95% confidence level (Source: https://abtestguide.com/abtestsize/)

You test a few changes together.

So what do you do?

“Are we in the business of science or the business of making money?”

Ensure each change addresses a specific problem

Three step sign-up flow
Control experience with existing plans selection page

Vs.

Test experience with multiple changes made on plans selection page to solve one customer problem

All changes support the same hypothesis

Control experience with existing pricing, plan selection and payment pages

Vs.

Test experience with new pricing, plan selection and payment pages

Additional Resources:

On the path to learning all things insights and optimization (linkedin.com/in/bithikamehra) | Foodie | Environmentalist| Loves to travel | Player of a few riffs

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store