Statistics Basics for A/B Testing: Part 2

In my last post, I covered some basic concepts important for A/B testing such as population, sample, measures of central tendency, standard deviation and confidence intervals. I will continue discussing other statistical concepts important to A/B testing in this post.

A hypothesis refers to the researcher’s initial belief about the situation before the study. This initial theory is known as the alternative hypothesis and the opposite is known as the null hypothesis. The null hypothesis is typically the “accepted fact”.

Hypothesis testing allows us to determine which theory, the null or alternative, is better supported by the evidence. So, we are basically testing whether the results are valid by figuring out the odds that the results have happened by chance. If the results have happened by chance, the experiment won’t be repeatable and so has little use.

Hypothesis testing is an important method of statistical inference and is widely used in a variety of studies — from medical trials to assess drug effectiveness to observational studies evaluating exercise plans to even randomized controlled experiments (aka A/B testing). What all studies have in common is that they are concerned with making comparisons, either between two groups or between one group and the entire population.

Continuing with the coffee cups temperature example from my previous post, let’s say that Starbucks makes the claim that their coffee is hotter than McDonalds. Given what we know about hypotheses statements, our hypotheses in this case will be:

Alternative Hypothesis: Starbucks coffee is hotter than McDonalds.

Null Hypothesis: Starbucks coffee is not hotter than McDonalds.

Since we can’t measure all the coffee cups at both places without some expensive fancy gizmo, we will use samples and hence, hypothesis testing to confirm if this is really the case.

We take a sample of 50 cups of coffee each from Starbucks and McDonalds. That sure is a lot of caffeine in our imaginary test! When we plot the temperatures of each of the coffee cups from Starbucks, we get a histogram that resembles a bell.