A/B Testing: Types of Tests
The type of A/B testing is generally a function of the type of data that is collected.
Hypothesis Testing
In all of these tests, there's a null hypothesis (H0) and an alternative hypothesis(Ha). The null hypothesis states that there is no difference in the outcomes. Basically, in other words, group A and group B have no difference. While the alternative hypothesis provides evidence that there is a difference in the outcomes.
Hypothesis Testing: One sample Test
Here, we have one set of data, and we want to know if the mean of this group is different from zero. The null hypothesis states that there is no different from zero, and the alternative hypothesis says that the group mean is different from zero.
Formula:
where,
X bar = sample mean
m = population mean
s = estimate of standard deviation of the population
n = sample size
Assumptions:
Hypothesis Testing: Equal sample sizes, equal variance
Conditions:
where,
= pooled standard deviation
(x bar)A, SA2 = mean and sample variance of control group (=group A)
n = nA = nB = sample size
Degree of freedom, v = 2n -2
Hypothesis Testing: Welch’s t-test
Condition:
Formula:
Hypothesis Testing: Paired sample t-test
Conditions:
What is meant by a paired sample?
Same sample is tested before and after a treatment.
For example:
Degree of freedom, v = n - 1
Hypothesis Testing
In all of these tests, there's a null hypothesis (H0) and an alternative hypothesis(Ha). The null hypothesis states that there is no difference in the outcomes. Basically, in other words, group A and group B have no difference. While the alternative hypothesis provides evidence that there is a difference in the outcomes.
Hypothesis Testing: One sample Test
Here, we have one set of data, and we want to know if the mean of this group is different from zero. The null hypothesis states that there is no different from zero, and the alternative hypothesis says that the group mean is different from zero.
Formula:
where,
X bar = sample mean
m = population mean
s = estimate of standard deviation of the population
n = sample size
Assumptions:
- The sample mean (x bar) follows a normal distribution with mean m and standard deviation s/√n
- The sample variance s2 follows a χ2 distribution with n-1 degrees of freedom
Hypothesis Testing: Equal sample sizes, equal variance
Conditions:
- Sample sizes of the two groups are equal: nA = nB
- The two populations have the same variance
where,
= pooled standard deviation
(x bar)A, SA2 = mean and sample variance of control group (=group A)
n = nA = nB = sample size
Degree of freedom, v = 2n -2
Hypothesis Testing: Welch’s t-test
Condition:
- Sample sizes of the two groups are either equal or unequal
- Two populations have normal distribution with unequal variances
Formula:
Hypothesis Testing: Paired sample t-test
Conditions:
- Samples are dependent
- Either one sample has been tested twice or two samples are paired
What is meant by a paired sample?
Same sample is tested before and after a treatment.
For example:
- Students before and after being taught for a test
- Pre-advertising and post-advertising purchase behaviour or opinion of individuals
Degree of freedom, v = n - 1
