Experiment Design: Controlling for Experimental Errors
Experimental error is the variation in the responses among the experimental units, under the same treatment and under the same conditions.
What is experimental error caused by?
How do researchers control some potential of experimental errors?
Experimental Procedures
The conditions under which the experiments are run must be nearly as constant as possible during the experiment à In the web page example, you might want to the degree possible control for whether some people might like to use the Internet when the weather it's not as nice outside. So if one city happens to have a lot of rain such as the Northwest, they might be endorsed more often looking at these web pages.
If the experimental procedures are not strictly followed the variance of the response can be inflated and the precision of our inferences or confidence intervals can be compromised.
Experimental procedure should be conducted uniformly for the duration of the experiment. Otherwise, a bias or inflated variance of the treatment may occur. A bias is a consistent overestimate or underestimate of the statistic that you're looking at.
Selecting Experimental and Measurement Units
Experimental error variance may increase if the experimental units used in the experiment are not similar with respect to those characteristics. Note, if the experimental units are overly uniform, then the generalizations to the population inferences may be restricted. For example, suppose a marketing firm wants to analyse whether different marketing techniques, such as a web ad, will affect the attention of students. Selecting students from the same grade level or school system as experimental units may achieve a more homogenous set of measurement units; but inferences from the experiment are highly limited because the result may be affected by grade level or school system.
Randomization of Treatments
Some of statistical procedures are based on the condition that the data were drawn from a population that was distributed normal. However, your actual sampling of the data may not follow the actual population distribution. We thus need to randomize the treatments.
Process:
Suppose we have ‘N’ experimental units and ‘t’ treatments. We want to randomly assign the ‘ith’ treatment to ‘ri’ experimental units. Note that r is the replication number of replications per experimental unit. So, r1 + r2 + … + ri = N. Completely randomized designs follow the steps:
1. Number the experimental units from 1 to N.
2. Randomly generate a list of numbers that is a random permutation of the numbers 1 to N.
3. Assign treatment #1 to the experimental units with the first r1 numbers in the list. Assign treatment #2 to the experimental units having the next r2 numbers in the list. Continue this step until all ‘t’ treatments are assigned.
What is experimental error caused by?
- Natural differences in the experimental units before receiving their allocation or the treatment.
- Differences in the experimental units before receiving their allocation or the treatment.
- Variation in the devices that record the measurements
- Variation in setting the treatment conditions
- Effect on the response variable of all extraneous factors other than the treatment factors
How do researchers control some potential of experimental errors?
- Experimental procedure à use of strict experimental procedures and following the procedures exactly the same way, every time so that we minimize experimental error in that dimension.
- Choice of experimental units and measurement units à If we are trying to perhaps measure how far away an object is if we measure in kilometres and round off to only whole numbers. We're going to get less accurate results than if we measured it by millimetres and we can get down to the millimetre level.
- Recording procedure à How we record the data? What kind of devices we use? Are they accurate devices?
- Type of experimental design
- Control variables
Experimental Procedures
The conditions under which the experiments are run must be nearly as constant as possible during the experiment à In the web page example, you might want to the degree possible control for whether some people might like to use the Internet when the weather it's not as nice outside. So if one city happens to have a lot of rain such as the Northwest, they might be endorsed more often looking at these web pages.
If the experimental procedures are not strictly followed the variance of the response can be inflated and the precision of our inferences or confidence intervals can be compromised.
Experimental procedure should be conducted uniformly for the duration of the experiment. Otherwise, a bias or inflated variance of the treatment may occur. A bias is a consistent overestimate or underestimate of the statistic that you're looking at.
Selecting Experimental and Measurement Units
Experimental error variance may increase if the experimental units used in the experiment are not similar with respect to those characteristics. Note, if the experimental units are overly uniform, then the generalizations to the population inferences may be restricted. For example, suppose a marketing firm wants to analyse whether different marketing techniques, such as a web ad, will affect the attention of students. Selecting students from the same grade level or school system as experimental units may achieve a more homogenous set of measurement units; but inferences from the experiment are highly limited because the result may be affected by grade level or school system.
Randomization of Treatments
Some of statistical procedures are based on the condition that the data were drawn from a population that was distributed normal. However, your actual sampling of the data may not follow the actual population distribution. We thus need to randomize the treatments.
Process:
Suppose we have ‘N’ experimental units and ‘t’ treatments. We want to randomly assign the ‘ith’ treatment to ‘ri’ experimental units. Note that r is the replication number of replications per experimental unit. So, r1 + r2 + … + ri = N. Completely randomized designs follow the steps:
1. Number the experimental units from 1 to N.
2. Randomly generate a list of numbers that is a random permutation of the numbers 1 to N.
3. Assign treatment #1 to the experimental units with the first r1 numbers in the list. Assign treatment #2 to the experimental units having the next r2 numbers in the list. Continue this step until all ‘t’ treatments are assigned.