![]() For example, it is used by Adobe Target 2 for A/B testing. The NHST approach has its benefits: straightforward formulas, clear decision thresholds, and almost a century of use across wide-ranging disciplines. that there is a difference between webpage variants). that there is no difference between webpage variants) and p-values less than 0.05 are taken to mean that the null hypothesis can be rejected (e.g. Conventionally, p-values greater than 0.05 are taken to mean that the null hypothesis cannot be rejected (e.g. Once the experiment has been run, then conclusions about it are based on theĬalculations of the p-value, which is the probability of seeing data as extreme as the experiment’s results, given the experimental conditions. The experiment needs to adhere to this pre-calculated sample size to be valid. how different the conversion rates will be between webpage variants). To calculate the required sample size, the experimenter must make some assumptions about the expected effect size (e.g. will detect the difference between variants if there is any). Before the experiment is run, the sample size needs to be determined, to ensure that the experiment will have enough power (i.e. The goal of a NHST experiment is to determine whether or not we can reject the null hypothesis (thereby accepting the alternate hypothesis). the two variants of the webpage will yield different conversion rates) are defined. the two variants of the webpage will yield the same conversion rate) and an alternate hypothesis (e.g. concluding that one variant has a better conversion rate than the other). The goal of a NHST experiment is to determine whether or not we can reject the null hypothesis (e.g. These decisions have largely been driven by using an approach known as Null-Hypothesis Significance Testing (NHST) (e.g. choosing a variant of a webpage that yields the highest conversion rate). Often in business settings, the goal is to pick the best performing variant from a set of options (e.g. If you use a product like Optimizely or Adobe Target, you are relying on these methods, whether you realise it or not. clinical trials in medicine) to the digital domain (e.g. The field of online experimentation 1 has grown immensely over the past decade, by applying methods from scientific research (e.g. In this post, we review the current trend of moving towards different approaches to experiment analysis. However, more recent work has highlighted shortcomings of this approach. Traditionally, such online experimentation has relied on Null-Hypothesis Significance Testing (NHST) to choose winning variants. Clear and unbiased analysis methods are critical to understanding the experiments that businesses run in-market on real users. ![]()
0 Comments
Leave a Reply. |