A/B Testing: Introduction
A/B Testing is a process of comparing two groups – test group and control group.
Two groups usually means two versions of some market asset. A market asset is something you want to research such as, if it's a webpage, a title, a layout of a design, a new marketing slogan or a phrase, a feature like the colour of a website button, a font, all these things are compared.
The only difference between the two groups is that the test group has the treatment, and the control group does not. Test treatment is the variation you want to test.
The test group has a change in only one element, such as a bold title vs normal title: A/B Testing vs A/B Testing.
Use case of A/B Testing: Digital Marketing
A/B Tetsing is used in digital marketing such as web page experiments. It allows us to test which feature achieves our goals better. For example, if we want to promote an ad and are designing a “Learn more” button, should we use a blue or a green button? Which one attracts more clicks?
We can test multiple variables, but only one at a time.
In the “blue or green button” example, the goal of changing the colour is to encourage the users to click. The metric here is the total number of clicks.
The following propositions are being assesses:
1. If X then Y (if the button is green, clicks will increase)
2. If not X then not Y (if the button is not green, clicks will not increase)
This proves the cause of effectiveness.
We have two equivalent groups and randomly assign subjects into the control and test groups. The control group is group A and the test group is group B.
Sample size is a key variable to determine the statistical significance level. Larger sample size à higher confidence level. Thus, before the test, sample size should be determined.
Common statistical significance level = 5% and confidence level = 95%.
The formula to determine the sample size of each group is:
Where,
n = number of subjects in each group
Z (a/2) = critical value of significance criterion
= significance level
S2 = sample estimate of variance. Rough estimate is d = (largest value – smallest value)/4
E = effective size
Two groups usually means two versions of some market asset. A market asset is something you want to research such as, if it's a webpage, a title, a layout of a design, a new marketing slogan or a phrase, a feature like the colour of a website button, a font, all these things are compared.
The only difference between the two groups is that the test group has the treatment, and the control group does not. Test treatment is the variation you want to test.
The test group has a change in only one element, such as a bold title vs normal title: A/B Testing vs A/B Testing.
Use case of A/B Testing: Digital Marketing
A/B Tetsing is used in digital marketing such as web page experiments. It allows us to test which feature achieves our goals better. For example, if we want to promote an ad and are designing a “Learn more” button, should we use a blue or a green button? Which one attracts more clicks?
- Procedure of A/B Testing
- Select test treatment
We can test multiple variables, but only one at a time.
- Identify your goal/ determine a metric
In the “blue or green button” example, the goal of changing the colour is to encourage the users to click. The metric here is the total number of clicks.
- Create test & control group
- Split your sample groups equally and randomly
The following propositions are being assesses:
1. If X then Y (if the button is green, clicks will increase)
2. If not X then not Y (if the button is not green, clicks will not increase)
This proves the cause of effectiveness.
We have two equivalent groups and randomly assign subjects into the control and test groups. The control group is group A and the test group is group B.
- Determine the sample size
Sample size is a key variable to determine the statistical significance level. Larger sample size à higher confidence level. Thus, before the test, sample size should be determined.
Common statistical significance level = 5% and confidence level = 95%.
The formula to determine the sample size of each group is:
Where,
n = number of subjects in each group
Z (a/2) = critical value of significance criterion
= significance level
S2 = sample estimate of variance. Rough estimate is d = (largest value – smallest value)/4
E = effective size
- Collect data and data analysis
- Ensure enough time to obtain substantial data
- Avoid conflicts between your test and other team’s test, project, or the company’s business plan
- Have a comprehensive data storage and back-up plan to ensure reproducibility
- Hypothesis Testing