JBDON
  • Home
  • Applied Analytics
    • Analytics for Decision Making >
      • What is Cluster Analysis
      • Data Reduction and Unsupervised Learning
      • Preparing Data and Measuring Dissimilarities
      • Hierarchical and k-Means Clustering
      • Defining Output Variables and Analyzing the Results
      • Using Historical Data to Model Uncertainty
      • Models with Correlated Uncertain Variables
      • Creating and Interpreting Charts
      • Using Average Values versus Simulation
      • Optimization and Decision Making
      • Formulating an Optimization Problem
      • Developing a Spreadsheet Model
      • Adding Optimization to a Spreadsheet Model
      • What-if Analysis and the Sensitivity Report
      • Evaluating Scenarios and Visualizing Results to Gain Practical Insights
      • Digital Marketing Application of Optimization
      • Advanced Models for Better Decisions
      • Business Problems with Yes/No Decisions
      • Formulation and Solution of Binary Optimization Problems
      • Metaheuristic Optimization
      • Chance Constraints and Value At Risk
      • Simulation Optimization
    • Analytics for Marketing >
      • Marketing Analytics and Customer Satisfaction
      • Customer Satisfaction
      • Measurements and Scaling Techniques – Introduction
      • Primary Scales of Measurement
      • Comparative Scaling
      • Non-Comparative Scaling
      • Experiment Design: Controlling for Experimental Errors
      • A/B Testing: Introduction
      • A/B Testing: Types of Tests
      • ANOVA – Introduction
      • Example -Inspect Spray and Tooth Growth
      • Logit Model - Binary Outome and Forecastign linear regression
      • Text Summarization
      • Social media Microscope
      • N-Gram - Frequcy Count and phase mining
      • LDA Topic Modeling
      • Machine-Learned Classification and Semantic Topic Tagging
    • Data Engine >
      • Understanding The Growth Of Data
      • Evaluating Methods Of Data Access
      • Communication journey
      • Data Journey
      • Planning for data visualisation
      • Visualisation Component
      • Content Connection and Chart Legitibility
    • Customer Insights >
      • Introduction
      • What is Descriptive Analytics?
      • Survey Overview
      • Net Promoter Score and Self-Reports
      • Survey Design
      • Passive Data Collection
      • Media Planning
      • Data Visualization
      • Causal Data Collection and Summary
      • Asking Predictive Questions
      • Regression Analysis
      • Data Set Predictions
      • Probability Models
      • Results and Predictions
      • Perspective Analytics (Maximize Revenue and Market Structure Competitions)
    • Analytics for Advance Marketing >
      • Visualisation and statistics (Political Advertising,Movie Theater and Data Assembly)
      • Excel Analysis of Motion Picture Industry Data
      • Displaying Conditional Distributions
      • Analyzing Qualitative Variables
      • Steps in Constructing Histograms
      • Common Descriptive Statistics for Quantitative Data
      • Regression-Based Modeling
      • Customer Analytics
      • Illustrating Customer Analytics in Excel
      • Customer Valuation Excel Demonstration
  • Soft Skills
    • Adaptability
    • Confidence
    • Change Management
    • Unlearning and Learning
    • Collaboration and Teamwork
    • Cultural Sensitivity
  • Marketing
  • Finance
  • Economics
    • Introduction to Managerial Economics >
      • Basic Techniques
      • The firm: Stakeholders, Objectives and Decision Issues
      • Demand and Revenue Analysis >
        • Demand Estimation and Forecasting
        • Demand Elasticity
        • Demand Concepts and Analysis >
          • Formulation and Solution of Binary Optimization Problems
      • Scope of Managerial Economics
    • Prodution and Cost Analysis >
      • Production Function
      • Estimation of Production and Cost Functions
      • Cost Concepts and Analysis I
      • Cost Concepts and Analysis II
    • Pricing Decisions >
      • Pricing strategies >
        • Adding Optimization to a Spreadsheet Model
      • Market structure and microbes barriers to entry
      • Pricing under pure competition and pure monopoly
      • Pricing under monopolistic and oligopolistic competition
    • Narendra Modi Development Model of Gujarat
  • JBDON Golf
    • Digital Marketing Application of Optimization
  • Let's Talk
  • MBA Project Sharing
  • About Us
    • Good Read >
      • IIMC says PepsiCo CEO Indra Nooyi was an average student
      • India’s middle class figures in Fortune’s Top Ten list of those who matter
      • The Start-Up of you.
      • BUYING AND MERCHANDISING
      • HUMAN RESOURCE MANAGEMENT
      • Do You Suffer From Decision Fatigue?
      • New Page
      • About social media and web 2.0
      • Building Your Own Start-up Technology Company, Part 1
      • Building Your Own Start-up Technology Company, Part 2
      • Building Your Own Start-up Technology Company, Part 3
      • Building Your Own Start-up Technology Company, Part 4
      • Renewable energy is no longer alternative energy
      • What Makes an Exceptional Social Media Manager?
      • The Forgotten Book that Helped Shape the Modern Economy
      • Home
      • How to Think Creatively
      • A Lighthearted Looks at Project Management and Sports Analogies
      • Why Trust Matters More Than Ever for Brands
  • CET Knowledge Zone
    • Tips From JBIMS Students >
      • Prasad Sawant
      • Chandan Roy
      • Ram
      • Ashmant Tiwari
      • Rajesh Rikame
      • Ami Kothari
      • Ankeet Adani
      • Sonam Jain
      • Marketing Analytics and Customer Satisfaction
      • Mitesh Thakker
      • Tresa Sankoorikal
    • Speed Techniques
    • CET Workshops
  • Untitled
  • New Page
    • Cluster analysis using excel and excel miner
    • Chance Constraints and Value At Risk
    • Adding Uncertainty to a Spreadsheet Model
  • Adidas

​A/B Testing: Introduction

Picture
A/B Testing is a process of comparing two groups – test group and control group.
Two groups usually means two versions of some market asset. A market asset is something you want to research such as, if it's a webpage, a title, a layout of a design, a new marketing slogan or a phrase, a feature like the colour of a website button, a font, all these things are compared.
The only difference between the two groups is that the test group has the treatment, and the control group does not. Test treatment is the variation you want to test.
The test group has a change in only one element, such as a bold title vs normal title: A/B Testing vs A/B Testing.
Use case of A/B Testing: Digital Marketing
A/B Tetsing is used in digital marketing such as web page experiments. It allows us to test which feature achieves our goals better. For example, if we want to promote an ad and are designing a “Learn more” button, should we use a blue or a green button? Which one attracts more clicks?
 









  • Procedure of A/B Testing
  • ​

  • Select test treatment
In this step, we select one independent variable that we want to test. In the “blue or green button” example, the colour of “Learn more” button is the only variable.
We can test multiple variables, but only one at a time.

  • Identify your goal/ determine a metric
We need to identify the goal before running the test as it will define the metric to assess the test. The metric will also be the dependent variable in our test.
In the “blue or green button” example, the goal of changing the colour is to encourage the users to click. The metric here is the total number of clicks.

  • Create test & control group
  • Split your sample groups equally and randomly
In a true experiment, we would want to create the test and control groups using random assignment.
The following propositions are being assesses:
1. If X then Y (if the button is green, clicks will increase)
2. If not X then not Y (if the button is not green, clicks will not increase)
This proves the cause of effectiveness.
 
We have two equivalent groups and randomly assign subjects into the control and test groups. The control group is group A and the test group is group B.

  • Determine the sample size
A/B Testing uses statistical significance to determine whether the test treatment contributes to the difference in metrics between the test group and the control group.
Sample size is a key variable to determine the statistical significance level. Larger sample size à higher confidence level. Thus, before the test, sample size should be determined.
Common statistical significance level = 5% and confidence level = 95%.
The formula to determine the sample size of each group is:
 
Where,
n = number of subjects in each group
Z (a/2) = critical value of significance criterion
 = significance level
S2 = sample estimate of variance. Rough estimate is d = (largest value – smallest value)/4
E = effective size

  • Collect data and data analysis
  • Ensure enough time to obtain substantial data
  • Avoid conflicts between your test and other team’s test, project, or the company’s business plan
  • Have a comprehensive data storage and back-up plan to ensure reproducibility
 
  • Hypothesis Testing

Powered by Create your own unique website with customizable templates.
  • Home
  • Applied Analytics
    • Analytics for Decision Making >
      • What is Cluster Analysis
      • Data Reduction and Unsupervised Learning
      • Preparing Data and Measuring Dissimilarities
      • Hierarchical and k-Means Clustering
      • Defining Output Variables and Analyzing the Results
      • Using Historical Data to Model Uncertainty
      • Models with Correlated Uncertain Variables
      • Creating and Interpreting Charts
      • Using Average Values versus Simulation
      • Optimization and Decision Making
      • Formulating an Optimization Problem
      • Developing a Spreadsheet Model
      • Adding Optimization to a Spreadsheet Model
      • What-if Analysis and the Sensitivity Report
      • Evaluating Scenarios and Visualizing Results to Gain Practical Insights
      • Digital Marketing Application of Optimization
      • Advanced Models for Better Decisions
      • Business Problems with Yes/No Decisions
      • Formulation and Solution of Binary Optimization Problems
      • Metaheuristic Optimization
      • Chance Constraints and Value At Risk
      • Simulation Optimization
    • Analytics for Marketing >
      • Marketing Analytics and Customer Satisfaction
      • Customer Satisfaction
      • Measurements and Scaling Techniques – Introduction
      • Primary Scales of Measurement
      • Comparative Scaling
      • Non-Comparative Scaling
      • Experiment Design: Controlling for Experimental Errors
      • A/B Testing: Introduction
      • A/B Testing: Types of Tests
      • ANOVA – Introduction
      • Example -Inspect Spray and Tooth Growth
      • Logit Model - Binary Outome and Forecastign linear regression
      • Text Summarization
      • Social media Microscope
      • N-Gram - Frequcy Count and phase mining
      • LDA Topic Modeling
      • Machine-Learned Classification and Semantic Topic Tagging
    • Data Engine >
      • Understanding The Growth Of Data
      • Evaluating Methods Of Data Access
      • Communication journey
      • Data Journey
      • Planning for data visualisation
      • Visualisation Component
      • Content Connection and Chart Legitibility
    • Customer Insights >
      • Introduction
      • What is Descriptive Analytics?
      • Survey Overview
      • Net Promoter Score and Self-Reports
      • Survey Design
      • Passive Data Collection
      • Media Planning
      • Data Visualization
      • Causal Data Collection and Summary
      • Asking Predictive Questions
      • Regression Analysis
      • Data Set Predictions
      • Probability Models
      • Results and Predictions
      • Perspective Analytics (Maximize Revenue and Market Structure Competitions)
    • Analytics for Advance Marketing >
      • Visualisation and statistics (Political Advertising,Movie Theater and Data Assembly)
      • Excel Analysis of Motion Picture Industry Data
      • Displaying Conditional Distributions
      • Analyzing Qualitative Variables
      • Steps in Constructing Histograms
      • Common Descriptive Statistics for Quantitative Data
      • Regression-Based Modeling
      • Customer Analytics
      • Illustrating Customer Analytics in Excel
      • Customer Valuation Excel Demonstration
  • Soft Skills
    • Adaptability
    • Confidence
    • Change Management
    • Unlearning and Learning
    • Collaboration and Teamwork
    • Cultural Sensitivity
  • Marketing
  • Finance
  • Economics
    • Introduction to Managerial Economics >
      • Basic Techniques
      • The firm: Stakeholders, Objectives and Decision Issues
      • Demand and Revenue Analysis >
        • Demand Estimation and Forecasting
        • Demand Elasticity
        • Demand Concepts and Analysis >
          • Formulation and Solution of Binary Optimization Problems
      • Scope of Managerial Economics
    • Prodution and Cost Analysis >
      • Production Function
      • Estimation of Production and Cost Functions
      • Cost Concepts and Analysis I
      • Cost Concepts and Analysis II
    • Pricing Decisions >
      • Pricing strategies >
        • Adding Optimization to a Spreadsheet Model
      • Market structure and microbes barriers to entry
      • Pricing under pure competition and pure monopoly
      • Pricing under monopolistic and oligopolistic competition
    • Narendra Modi Development Model of Gujarat
  • JBDON Golf
    • Digital Marketing Application of Optimization
  • Let's Talk
  • MBA Project Sharing
  • About Us
    • Good Read >
      • IIMC says PepsiCo CEO Indra Nooyi was an average student
      • India’s middle class figures in Fortune’s Top Ten list of those who matter
      • The Start-Up of you.
      • BUYING AND MERCHANDISING
      • HUMAN RESOURCE MANAGEMENT
      • Do You Suffer From Decision Fatigue?
      • New Page
      • About social media and web 2.0
      • Building Your Own Start-up Technology Company, Part 1
      • Building Your Own Start-up Technology Company, Part 2
      • Building Your Own Start-up Technology Company, Part 3
      • Building Your Own Start-up Technology Company, Part 4
      • Renewable energy is no longer alternative energy
      • What Makes an Exceptional Social Media Manager?
      • The Forgotten Book that Helped Shape the Modern Economy
      • Home
      • How to Think Creatively
      • A Lighthearted Looks at Project Management and Sports Analogies
      • Why Trust Matters More Than Ever for Brands
  • CET Knowledge Zone
    • Tips From JBIMS Students >
      • Prasad Sawant
      • Chandan Roy
      • Ram
      • Ashmant Tiwari
      • Rajesh Rikame
      • Ami Kothari
      • Ankeet Adani
      • Sonam Jain
      • Marketing Analytics and Customer Satisfaction
      • Mitesh Thakker
      • Tresa Sankoorikal
    • Speed Techniques
    • CET Workshops
  • Untitled
  • New Page
    • Cluster analysis using excel and excel miner
    • Chance Constraints and Value At Risk
    • Adding Uncertainty to a Spreadsheet Model
  • Adidas