Statistics Basics for A/B Testing: Part 2

In my last post, I covered some basic concepts important for A/B testing such as population, sample, measures of central tendency, standard deviation and confidence intervals. I will continue discussing other statistical concepts important to A/B testing in this post.

A hypothesis refers to the researcher’s initial belief about the situation before the study. This initial theory is known as the alternative hypothesis and the opposite is known as the null hypothesis. The null hypothesis is typically the “accepted fact”.

Hypothesis testing allows us to determine which theory, the null or alternative, is better supported by the evidence. So, we are basically testing whether the results are valid by figuring out the odds that the results have happened by chance. If the results have happened by chance, the experiment won’t be repeatable and so has little use.

Hypothesis testing is an important method of statistical inference and is widely used in a variety of studies — from medical trials to assess drug effectiveness to observational studies evaluating exercise plans to even randomized controlled experiments (aka A/B testing). What all studies have in common is that they are concerned with making comparisons, either between two groups or between one group and the entire population.