Member-only story
Statistics Basics for A/B Testing: Part 3
In the last few posts, I have introduced basic statistical concepts you need to know for A/B testing. In the last post of this series, I will be covering the concept of statistical power, types of errors and a brief on sample size calculation.
In my last post, we were able to reject the null hypothesis that “Starbucks coffee is not hotter than McDonalds” since the observed p-value of our z-test was lower than the 0.05 p-value we had set apriori.
We also learned that the p-value is the probability of observing results at least as extreme as those measured when the null hypothesis is true or due to random chance. If the observed p-value is less than alpha, then the results are statistically significant i.e. the probability of seeing these results due to random chance is very low.
When we set a 5% statistical significance level (or a 95% confidence level), and if the observed p-value is less than that, it simply means that the probability of observing these results due to random chance is less than 5 times if we were to repeat the experiment 100 times. So, the p-value essentially sets a threshold for the false positive rate that the experimenter or the researcher is…
