Module 5: Validity of Experimental Design

Essential questions 

What are the internal and external validity of experimental design?

What are the threats to internal validity and how to control them? 

To produce reliable outcomes and statistically relevant testing of hypothesis, experimental design must satisfy the parameters of internal and external validity.

Internal validity of the experimental design is the degree to which the observed outcomes reflect the effect of a specific treatment that was applied to the control group, instead of being a result of extraneous variables.

External validity of the experimental design is the extent to which the results can be extrapolated or applied to other specific populations, treatment options, measurement variables, etc. In other words, to what extent the observed outcome can be generalized.

Internal validity can be compromised by the following factors:

• History, or events that happen in the experimental system between the two successive measurements of the treatment outcomes. 
• Maturation, which is the result of the biological subjects getting tired, older, or similar changes.
• Repeated testing that exposes the subject to the first measurement so that it has a prior experience that can affect the outcome of the second experimental measurement.
• Differential selection of individuals or groups so that some of them could have had a prior knowledge or the ability that alters the measured outcome.
• Experimental mortality – the change in numbers of subjects in experimental or control group over the course of experiment so that the unique contribution of those individuals in the measured outcomes is lost.
• Measuring instruments – the changes in their calibration, sensitivity, and operators can affect the outcomes of measurements.
• Confounding: changes in the dependent variable are not due to an independent variable but to another factor that has effect on both independent variable and dependent variable.  

These factors are changing the outcomes of the measurements of dependent variable and affect the causal relationship between the treatment and outcomes. Therefore, they may lead to the erroneous conclusions regarding the validity of hypothesis that was tested statistically.

Use this video:

Suggested readings

Rossi P.H., Wright J. D. (1984) Evaluation Research: An Assessment. Annual Review of Sociology. 10:331-352.