A/B Test Process
Hypothesis
→Design
→Run
→Analyze
What is A/B Testing?
A/B testing (split testing) compares two versions to determine which performs better. It's the gold standard for data-driven decisions.
- Control (A): Current version
- Treatment (B): Modified version
- Random assignment: Users see A or B
The A/B Testing Process
- Form Hypothesis: "Changing X will increase Y because Z"
- Determine Metrics: Primary, secondary, guardrail
- Calculate Sample Size: Before starting
- Run Experiment: Random assignment, don't peek!
- Analyze Results: Statistical significance
Key Statistical Concepts
| Concept | Definition | Typical |
|---|---|---|
| Significance (α) | False positive probability | 5% |
| Power (1-β) | Detect true effect | 80% |
| p-value | Chance of result | < 0.05 |
Common Pitfalls
- Peeking: Stopping early when you see desired result
- Multiple comparisons: Testing many variants
- Novelty effect: Users react to newness
- Insufficient sample: Underpowered tests
Conclusion
Key Takeaways
- A/B testing: Control vs. Treatment
- Hypothesis first, then test
- Don't peek - run full duration
- p < 0.05 = statistically significant
- Consider practical significance too