Statistics Basics for A/B Testing: Part 1
Knowledge of basic statistics is extremely important in designing and analyzing A/B tests accurately. Over the next few posts, I will be reviewing some important concepts all conversion optimization professionals should know and apply.
We need statistics any time it isn’t feasible to study the whole population to understand their attitudes, opinions, preferences etc. Let me explain with an example. Suppose we are interested in knowing which place in town has hotter coffee — McDonald’s or Starbucks. It would be difficult to measure each and every cup of coffee at both these places without some fancy gadgets.
So what we do instead is take a few cups of coffee (i.e. a sample) from both McDonald’s and Starbucks and measure their temperature. This data from both the samples can then be used to infer which place serves hotter coffee.
So, Population is the entire pool of users or things we want to measure. For a website, it is all your website visitors. When we A/B test, we essentially study a sample of visitors in experience A and a sample of visitors in experience B to make inference about the population. It can be to see if a new homepage design or a different checkout…
