In this module, internal and external validity will be explained and threats to validity will be discussed.

Learning Objectives

• Define internal and external validity
• List and explain threats to internal validity
• List and explain threats to external validity

Begin this module by watching the video, Validity, that describes internal and external validity, discusses threats to validity and offers examples. This video is a great introduction to the topic of validity in experimental research. Click here to watch the video.

As seen in the video, there are two types of validity: internal validity and external validity. Internal validity refers to the validity of the findings within the research study. It is primarily concerned with controlling the extraneous variables and outside influences that may impact the outcome. This is especially important in experimental studies to ensure that the experimental treatment (X) is, in fact, responsible for a change in the dependent variable (Y). This is critical if the study is going to be able to determine a causal relationship. Therefore, the researcher must plan to control or eliminate the influence of other variables to be confident when making conclusions about the relationship between X and Y. For example, if a researcher wanted to determine if there was a causal relationship between increasing physical activity and lowering cholesterol levels, he or she would need to consider other factors that impact cholesterol levels and attempt to eliminate those influences in the test group.

External validity refers to the extent to which the results of study can be generalized or applied to other members of the larger population being studied. For this reason, the random selection of participants and random assignment of the study participants into groups is critical so that the members of the study are truly representative of the larger population. External validity is concerned with real life applications that have relevance beyond the confines of the experiment. Random selection is really the key ensuring that results are generalizable. In the physical activity/cholesterol example, the researcher would want to be sure that race, gender, age, BMI, and other factors that may differ among test subjects did not influence the results. Again, random selection of participants should control for these influences.

Factors that have the potential to influence the findings or the generalizability of the findings are called threats to validity. Following are lists of threats to internal and external validity in experimental research:

Threats to Internal Validity

• History – This refers to unplanned events that may occur during a study that impact the results unintentionally. Test subjects often have different experiences as the study progresses that may have an influence. For example, if doing a pretest and a post-test assessment at the beginning and end of a semester for two different classrooms to compare test results, one group may have a different classroom atmosphere or dynamic that influences the post-test results.
• Maturation – Natural changes, biological or psychological, within the participants over the time of the study may impact the results. Test subjects may become bored, tired, hungry, and so forth during the time of the study. This is more of an issue with long-term studies.
• Testing – Experiments that pretest the subjects may influence the performance of subjects on subsequent tests simply since participants have already seen or completed the test before. People tend to perform better at any activity the more they are exposed to it.
• Instrumentation – Changes in testing instrumentation during a study may affect what is being measured and how it is measured. Similarly, if human observations are involved, the observations or perceptions of the of the observers may change over time, rather than the actual performance of the test subjects.
• Statistical Regression – Statistical regression, or regression to the mean, can be a concern in studies with extreme scores, either particularly high or low. Scores are typically not as extreme in subsequent testing in most situations, making meaningful pretest and post-test comparisons more difficult.
• Selection – If the subjects placed into the groups are selected in a non-random manner or are functionally inequivalent at the beginning of the study, the results of the study will be biased when making comparisons between the groups at the end.
• Experimental Mortality – Test subjects drop out of studies for a variety of reasons.   The loss of participants from comparison groups may impact the study if the withdrawal or mortality rate is higher in one group or if it is particularly high in both groups. For example, a larger number of participants may drop out of a group due to illness while several less motivated participants drop out of the other group. The groups no longer have a similar make up of individuals.
• Selection Interaction – The selection method may interact with one or more of the other threats and impact results. For example, groups with larger numbers of elderly participants may be impacted more by maturation during the study.

Threats to External Validity

• Interaction Effects of Testing – The pretest may make the participants more aware of or sensitive to the treatment that will be applied and therefore, may influence the response to the treatment.
• Selection Bias – This occurs when subjects are selected in a manner that does not ensure that they are representative of the overall population. The random selection of subjects is a critical factor in determining external validity.
• Reactive Effects of Experimental Testing – The fact that treatments in a controlled, laboratory setting may differ from those in a less controlled, real-world environment. The performance of the subjects may be more due to the setting than the independent variable.
• Multiple Treatment Interference – When subjects receive more than one treatment, the effects of previous treatments may influence responses. Early treatments may have a cumulative effect on how subjects perform or respond.

No experimental research project is perfect or free from potential threats to validity. Researchers must take the necessary steps to ensure that the threats are controlled as best as possible. The most common methods to achieve internal and external validity are randomization and the use of appropriate research designs and statistical analyses. Planning and foresight are important in controlling these threats when creating the experimental design. The next module will focus on common sources of error and bias, which are additional factors that may influence the findings.

Suggested Readings

Bernard, H. R., & Bernard, H. R. (2012). Social research methods: Qualitative and quantitative approaches. Sage.
Chen, H. T., & Rossi, P. H. (1987). The theory-driven approach to validity. Evaluation and program planning, 10(1), 95-103.
Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.
Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. Longman Publishing.
Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (1993). How to design and evaluate research in education (Vol. 7). New York: McGraw-Hill.
Keppel, G. (1991). Design and analysis: A researcher's handbook. Prentice-Hall, Inc.
Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research (Vol. 19). Sage.
Neuman, W. L., & Neuman, W. L. (2006). Social research methods: Qualitative and quantitative approaches.
Neuman, W. L., & Robson, K. (2004). Basics of social research. Pearson.
Punch, K. F. (2013). Introduction to social research: Quantitative and qualitative approaches. Sage.