A/B Testing: Introduction

A/B Testing is a process of comparing two groups – test group and control group. Two groups usually means two versions of some market asset. A market asset is something you want to research such as, if it's a webpage, a title, a layout of a design, a new marketing slogan or a phrase, a feature like the colour of a website button, a font, all these things are compared. The only difference between the two groups is that the test group has the treatment, and the control group does not. Test treatment is the variation you want to test. The test group has a change in only one element, such as a bold title vs normal title: A/B Testing vs A/B Testing. Use case of A/B Testing: Digital Marketing A/B Tetsing is used in digital marketing such as web page experiments. It allows us to test which feature achieves our goals better. For example, if we want to promote an ad and are designing a “Learn more” button, should we use a blue or a green button? Which one attracts more clicks?

• Procedure of A/B Testing

• Select test treatment

In this step, we select one independent variable that we want to test. In the “blue or green button” example, the colour of “Learn more” button is the only variable. We can test multiple variables, but only one at a time.

• Identify your goal/ determine a metric

We need to identify the goal before running the test as it will define the metric to assess the test. The metric will also be the dependent variable in our test. In the “blue or green button” example, the goal of changing the colour is to encourage the users to click. The metric here is the total number of clicks.

• Create test & control group

• Split your sample groups equally and randomly

In a true experiment, we would want to create the test and control groups using random assignment. The following propositions are being assesses:

• If X then Y (if the button is green, clicks will increase)
• If not X then not Y (if the button is not green, clicks will not increase)

This proves the cause of effectiveness. We have two equivalent groups and randomly assign subjects into the control and test groups. The control group is group A and the test group is group B.

• Determine the sample size

A/B Testing uses statistical significance to determine whether the test treatment contributes to the difference in metrics between the test group and the control group. Sample size is a key variable to determine the statistical significance level. Larger sample size à higher confidence level. Thus, before the test, sample size should be determined. Common statistical significance level = 5% and confidence level = 95%. The formula to determine the sample size of each group is: Where, n = number of subjects in each group Z (a/2) = critical value of significance criterion = significance level S2 = sample estimate of variance. Rough estimate is d = (largest value – smallest value)/4 E = effective size

• Collect data and data analysis

• Ensure enough time to obtain substantial data
• Avoid conflicts between your test and other team’s test, project, or the company’s business plan
• Have a comprehensive data storage and back-up plan to ensure reproducibility

• Hypothesis Testing