Statistics Basics for A/B Testing: Part 1

Knowledge of basic statistics is extremely important in designing and analyzing A/B tests accurately. Over the next few posts, I will be reviewing some important concepts all conversion optimization professionals should know and apply.

Source: Quora

We need statistics any time it isn’t feasible to study the whole population to understand their attitudes, opinions, preferences etc. Let me explain with an example. Suppose we are interested in knowing which place in town has hotter coffee — McDonald’s or Starbucks. It would be difficult to measure each and every cup of coffee at both these places without some fancy gadgets.

So what we do instead is take a few cups of coffee (i.e. a sample) from both McDonald’s and Starbucks and measure their temperature. This data from both the samples can then be used to infer which place serves hotter coffee.

So, Population is the entire pool of users or things we want to measure. For a website, it is all your website visitors. When we A/B test, we essentially study a sample of visitors in experience A and a sample of visitors in experience B to make inference about the population. It can be to see if a new homepage design or a different checkout flow perform better.

Going back to the coffee cups example, now that we have the data, what do we do? We need to find a way to compare the two samples. This is where we use “mean” which is a measure of central tendency. A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data. Mode and median are other measures of central tendency and when the data is skewed could be better measures of central tendency than mean.

When A/B testing, this could be your conversion rate, average order value or any other metric close to the point of change to measure it’s impact.

Once we plot the different temperatures in each of the samples, we get to something that looks like this.