This module explores potential sources of error and bias in experimental research.

Learning Objectives

• Define experimental error and bias
• Describe the types of error common in experimental research and provide examples
• Explain the different types of bias that may affect experimental research and provide examples

The following SlideShare presentations, Causes and Types of Experimental Error and Error, Bias, and Confounding, explain the various types of potential error and bias that may affect experimental research and provide numerous examples to demonstrate these concepts. Sources of experimental errors and potential biases are explained below in more detail.

Error, confounding and bias from Amandeep Kaur

Errors and Error Measurements from Milind Pelagade

Experimental Errors

No set of experimental data is perfect and, in general, researchers are aware that the data contains some margin of error. Many potential sources of error cause the results of a research study to interpret incorrectly. There are three basic types of errors that will be described below: human error, systematic error, and random error.

• Human Errors – This occurs when the researcher simply makes a mistake such as misreading an instrument or setting up the experiment wrong. This type of error is not “experimental” error, but rather an actual mistake. It does impact the findings, nonetheless.
• Systematic Errors – This type of error is caused by the way the experiment was conducted or the design of the experiment. Because these types of errors are inherent in the experimental set up, they skew the data consistently in one direction. For example, if the researcher was measuring the acceleration of the fall of an object due to gravity, air resistance would consistently slow down the fall of the objects and add to the fall time. Systematic errors can only be rectified by changing the experimental setup.
• Random Errors – Random errors are unpredictable and are chance variations in the measurements over which the researcher has little, if any, control. Random errors are equally as likely to be as they are to be low. This does help to reduce the impact of random error if the data set is large because the results can be “averaged”. For example, measurements made with a stopwatch may be slightly shorter or longer than the actual time.

There are some specific types of experimental error that effect the study and are threats to the validity. Following is list of those types of effects:

• Hawthorne Effect – This is a reactive effect in which the research participant does not behave normally because he or she knows that they are in a study. For example, it may change a person’s eating habits when they are part of a dietary study. It may not be possible for participants to be unaware of the study, but at a minimum, they should not be made aware of the hypothesized outcome.
• Placebo Effect – The placebo effect occurs when the participant believes the experimental treatment will cause a change. The participant may respond with a change in behavior or performance that is not actually due to the independent variable. This is commonly seen in drug trials. For example, a person takes a new arthritis medication and is convinced it helps them, but it may only be that they “believe” it helps.
• John Henry Effect – This is a threat to internal validity in that the participants in the control group try harder if they know they are in the control group.
• Rating Effect – The rating effect refers to the subjective nature of rating the participants. The rater will often rank some high and some low.

Biases

Bias is described as the intentional or unintentional influence that the researcher may have on a study. Therefore, a bias will prejudice the results of the research findings. In general, bias is a type of systematic error that is introduced into the sampling or testing and encourages one outcome over another. Some degree of bias is present in nearly all research projects. So, the question is not whether or not there is bias, but rather, the extent to which bias influenced the results. Following is a description of several types of common sources of bias that may occur in experimental research:

• Sampling Bias – This type of bias occurs when a bias is present in choosing the target population to sample and the participants are not representative of the larger population. For example, if a researcher was studying feelings on gun control, and the target area chosen for the study was an area that is predominantly made up of one political party or another, it may skew the results. The results cannot then be generalized to the larger population.
• Selection Bias – If the participants are not equally and randomly assigned to the experimental and control groups, selection bias will occur. It may also occur in the way that test subjects are chosen to participate. All members of the target population should have an equal chance of being selected and an equal chance being placed into any group.
• Response Bias – This occurs when only certain types of individuals respond to an invitation to participate. The participant group is then not representative of the larger population.
• Performance Bias – Performance bias occurs when something effects how the treatments or interventions are delivered. This typically occurs when the participants or the researchers behave differently because they are part of a study. For example, if a researcher feels that treatment A is more effective than treatment B, he or she may be more attentive to the participants receiving treatment A. Blinding is a common way to rectify this situation. In a single blind study, the participants do not know whether they are in the control or the experimental group, thereby eliminating any bias by participants. In a double-blind study, neither the participants nor the researcher knows which group is receiving the treatment.
• Measurement Bias – The people measuring or assessing the outcomes in a study should not know which individuals belonged to which groups to ensure that they do not focus only on data supporting the intervention.

This module focused on common sources of error and bias in experimental research studies. It is important to carefully plan the study to avoid this source of error and bias from the beginning of the project. This will ensure more confidence in the findings and higher validity.

Suggested Readings

Baum, W. M. (1974). On two types of deviation from the matching law: Bias and undermatching. Journal of the experimental analysis of behavior, 22(1), 231.
Bernard, H. R., & Bernard, H. R. (2012). Social research methods: Qualitative and quantitative approaches. Sage.
Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.
Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. Longman Publishing.
Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (1993). How to design and evaluate research in education (Vol. 7). New York: McGraw-Hill.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Sage.
Keppel, G. (1991). Design and analysis: A researcher's handbook. Prentice-Hall, Inc.
Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research (Vol. 19). Sage.
Neuman, W. L., & Neuman, W. L. (2006). Social research methods: Qualitative and quantitative approaches.
Punch, K. F. (2013). Introduction to social research: Quantitative and qualitative approaches. Sage.