This module discusses internal and external validity in quasi-experimental research studies and addresses threats to validity.

Learning Objectives

• Define internal and external validity
• Describe threats to validity in quasi-experimental research
• Discuss ways to address threats to validity and provide examples

There are two types of validity concerns: internal validity and external validity. Below is a link to a video, Validity, that provides an excellent introduction to the topic of validity concerns in research. The video describes internal and external validity, discusses threats to validity, and offers examples. Click here to watch the video. Validity concerns are very important in quasi-experimental research because the research designs lack the same level of control mechanisms as true experimental designs, thereby raising questions regarding the validity of the research findings.

Internal validity refers to the validity of the findings within the research study. It is primarily concerned with controlling the extraneous variables and outside influences that may impact the outcome. This is important in experimental studies and quasi-experimental studies that are attempting to demonstrate causation to ensure that the experimental treatment (X) is, in fact, responsible for a change in the dependent variable (Y). Internal validity is critical if the study is going to be able to determine a causal relationship. Therefore, the researcher must plan to control or eliminate the influence of other variables to be confident when making conclusions about the relationship between X and Y. For example, if a researcher wanted to determine if there was a causal relationship between increasing physical activity and lowering cholesterol levels, he or she would need to consider other factors that impact cholesterol levels and attempt to eliminate those influences in the test group. This is considerably more difficult in quasi-experimental studies than in experimental studies, primarily because participants are not randomly selected and therefore, it is more difficult to control the extraneous variables that may influence the findings.

External validity refers to the extent to which the results of study can be generalized or applied to other members of the larger population being studied. External validity is concerned with real life applications that have relevance beyond the confines of the experiment. In true experimental studies, the random selection of participants and random assignment of the study participants into groups ensures that the members of the study are truly representative of the larger population. Random selection is really the key ensuring that results are generalizable. In the physical activity/cholesterol example, the researcher would want to be sure that race, gender, age, BMI, and other factors that may differ among test subjects did not influence the results. Again, random selection of participants should control for these influences. Quasi-experimental studies present challenges because participants are NOT randomly selected. Construction of a reasonably similar control group is key, and the matching methods described in the previous module may be used help minimize this issue. The best evidence for external validity is whether the research findings can be reproduced with different populations, settings, or times. Researchers should keep replication in mind when designing the study to ensure external validity.

Factors that have the potential to influence the findings or the generalizability of the findings are called threats to validity. Following are lists of threats to internal and external validity.

Threats to Internal Validity

• History – This refers to unplanned events that may occur during a study that impact the results unintentionally. Test subjects often have different experiences as the study progresses that may have an influence. For example, if doing a pretest and a post-test assessment at the beginning and end of a semester for two different classrooms to compare test results, one group may have a different classroom atmosphere or dynamic that influences the post-test results.
• Maturation – Natural changes, biological or psychological, within the participants over the time of the study may impact the results. Test subjects may become bored, tired, hungry, and so forth during the time of the study. This is more of an issue with long-term studies.
• Testing – Experiments that pretest the subjects may influence the performance of subjects on subsequent tests simply since participants have already seen or completed the test before. People tend to perform better at any activity the more they are exposed to it.
• Instrumentation – Changes in testing instrumentation during a study may affect what is being measured and how it is measured. Similarly, if human observations are involved, the observations or perceptions of the of the observers may change over time, rather than the actual performance of the test subjects.
• Statistical Regression – Statistical regression, or regression to the mean, can be a concern in studies with extreme scores, either particularly high or low. Scores are typically not as extreme in subsequent testing in most situations, making meaningful pretest and post-test comparisons more difficult.
• Selection – If the subjects placed into the groups are selected in a non-random manner or are functionally inequivalent at the beginning of the study, the results of the study will be biased when making comparisons between the groups at the end.
• Experimental Mortality – Test subjects drop out of studies for a variety of reasons. The loss of participants from comparison groups may impact the study if the withdrawal or mortality rate is higher in one group or if it is particularly high in both groups. For example, a larger number of participants may drop out of a group due to illness while several less motivated participants drop out of the other group. The groups no longer have a similar make up of individuals.
• Selection Interaction – The selection method may interact with one or more of the other threats and impact results. For example, groups with larger numbers of elderly participants may be impacted more by maturation during the study.

Threats to External Validity

• Interaction Effects of Testing – The pretest may make the participants more aware of or sensitive to the treatment that will be applied and therefore, may influence the response to the treatment.
• Selection Bias – This occurs when subjects are selected in a manner that does not ensure that they are representative of the overall population. The random selection of subjects is a critical factor in determining external validity.
• Reactive Effects of Experimental Testing – The fact that treatments in a controlled, laboratory setting may differ from those in a less controlled, real-world environment. The performance of the subjects may be more due to the setting than the independent variable.
• Multiple Treatment Interference – When subjects receive more than one treatment, the effects of previous treatments may influence responses. Early treatments may have a cumulative effect on how subjects perform or respond.

No experimental research project is perfect or free from potential threats to validity, and it is even more important in quasi-experimental studies because the lack of random selection of test subjects creates uncertainty. Researchers must take the necessary steps to ensure that the threats are controlled as best as possible. Careful selection of the appropriate research design and attention to control methods is critical. Following are examples of threats to validity in the most common quasi-experimental designs (discussed in previous modules) and how those threats may be addressed.

• Pre-test/Post-test No Control Group Design – A researcher is studying the impact of a nutrition education program on a group of overweight teenagers and plans to weigh subjects prior to the program and then again after 6 months. If the group loses weight, can the researcher be sure it is due to the education program? Internal validity threats may include a maturation effect, a history effect and a testing effect. The best way to address these threats may be to add a control group to the design to compare results. This would provide additional control to ensure that something like a change in the school lunch program is not also contributing to the results. Perhaps growth spurts of teenagers at that age is contributing to the weight loss. A control group would allow for comparison with other teenagers to rule out the impact of these types of variables. A simple Pre-Test/Post-Test design with no control group typically creates the largest validity concerns.
• Non-equivalent Control Group Design – A researcher is looking at the incidence of depression in divorced individuals ages 40-50 years old. The incidence of depression in this group will need to be compared to the general population in that age category. The biggest threat to validity in this design is in the selection of the test subjects and creation of the control group. Matching methods will need to be used to create a comparison group that has similar demographics to ensure that the depression is not due to income levels, health conditions or other factors. If a valid control group can be created, the Non-equivalent Control Group Design is typically considered the strongest of the quasi-experimental design. While the groups may not be equivalent on potential variables, this design does control the other validity threats better than other quasi-experimental designs.
• Time-Series Designs – A study is being done to examine the impact of a new campus crime prevention program on new student enrollment. The researcher would collect several years worth of data regarding the crime rate and new student enrollment prior to the implementation of the program and several years of data following the implementation of the new program. The major threats to validity in this study are the history effect and perhaps maturation. The researcher would have to examine other campus events or activities that may be impacting new student enrollment during this time, but the Time-Series Design does control for other types of validity threats quite well.

The following YouTube video, Quasi-Experiments, provides a more detailed discussion regarding threats to validity in quasi-experimental research designs, with an emphasis on selecting the appropriate research design to maximize validity and reduce threats.

Suggested Readings

Bernard, H. R., & Bernard, H. R. (2012). Social research methods: Qualitative and quantitative approaches. Sage.

Brown, L. (2010). Quasi-experimental research. Doing Early Childhood Research: International perspectives on theory and practice, 345.

Campbell, D. T., & Stanley, J. C. (2015). Experimental and quasi-experimental designs for research. Ravenio Books.

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.

Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into practice, 39(3), 124-130.

Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research (Vol. 19). Sage.

Maxwell, J. (1992). Understanding and validity in qualitative research. Harvard educational review, 62(3), 279-301.

Punch, K. F. (2013). Introduction to social research: Quantitative and qualitative approaches. Sage.

William R.. Shadish, Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning.