Module 6: Critical Factors for External Validity

Essential questions 

What are the factors that affect external validity of experimental design?

How can they preclude generalization of research outcomes? 

External validity of the experimental design is the extent to which the results can be extrapolated or applied to other specific populations, treatment options, measurement variables, etc. In other words, to what extent the observed outcome can be generalized.

Video

External validity can be compromised by the following factors:

• Pre-testing: subjects that were previously tested, have prior experience with the treatment, and are not representative of the population that had not been pre-tested. This is especially true for humans as their impression upon pre-testing can affect the reported effects of the subsequent testing. 
• Differential selection of individuals or groups determines the generalizability of research outcomes as they may not be extrapolated on a subject group with different characteristics. Random selection of subjects from the general population creates a representative study group that can be generalized to the whole population. 
• Experimental procedures – the effect of experimental setting and instruments on the subjects may preclude generalization of research outcomes.
• Multiple treatment interference – the outcomes of the studies where multiple treatments were applied to a group of subjects, can only be generalized to the similar group that had experienced the same treatments in the same order. 
• Replications are the ultimate tests to the external validity of experiment. Replications can be done by the same researcher in the same experimental setting, or by different researchers in the same or different settings to verify that the outcome is still consistent with the underlying hypothesis that was confirmed by the original experiment. 

As the amount of similar experimental data increases, meta-analysis can be used to average the results of many experimental values and verify if the effect of independent variable is reliable or if the similar effect can be attributed to random chance.

Generalization of research outcomes and their applicability depend on successful minimization of the threats to external validity of experimental design. This expands the range of systems to which the experimentally obtained relationships can be extrapolated.

Suggested readings

Lynch J. (1999). Theory and External Validity. Journal of the Academy of Marketing Science. 27 (3): 367–76.

Mitchell M., Jolley J. (2001). Research Design Explained (4th Ed) New York: Harcourt.