Validity (Internal and External)
Validity is the set of criteria by which the credibility of research is measured. Validity is measured in degrees (ie. more or less valid).
(Garth Frazer, PPG1004 2008)
Internal validity: a study is internally valid if the statistical inferences about causal effects are valid for the population that is being studied. We ask the question: Is there any factor other than the independant variable, X, that could be partly responsible for the observed association between X and the dependant variable, Y?
Threats to Internal Validity (and Solutions):
1) Omitted Variable Bias (OVB)
OVB arises when an omitted variable is (1) a determinant of Y and (2) correlated with at least one regressor (X).
Solutions to OVB:
Randomized controlled experiment;
If the omitted variable is observable, include it in regression;
Use panel data in which each entity is observed more than once
If the variable cannot be measured, use the instrumental variables regression.
2) Wrong Functional From
Arises when the functional form is incorrect (ie. using linear instead of log). For example, when an interaction term is omitted, then inferences about causal effects will be biased.
Solutions to Wrong functional form:
Use appropriate functional form – use nonlinear specifications such as Logarithms interactions, polynomials, etc.
3) Errors in variable bias is the error in measurement.
Different types of error include:
data entry error
recall errors in surveys
asking ambiguous questions in surveys/ experiments
intentionally false responses to survey questions
Solutions to Errors in variable bias:
Collect better data
Develop specific model to measure error. This is only possible if we know a lot about the nature of the measurement error.
Instrumental variables regression.
4) Sample selection bias
Arises when something influences the availability of data and is related to the dependent variable, leading to a biased sample.
Example: If we were to compare the performance of actively managed mutual funds to passively managed mutual funds, we need to be sure that the sample is not biased. If we were to examine the perfomance of funds over the last 10 years, but only use as our sample, the funds available at the end of the ten year period, the funds that failed over the ten year period would not be accounted for. This will cause a bias in our analysis.
Solutions to sample selection bias:
Collect data that avoids sample selection (In the example of mutual funds, we could change the sample population to all those funds available at the beginning of the ten year period)
Randomized controlled experiment
5) Simultaneous causality bias
So far we have assumed that X causes Y but what if Y causes X as well?
Solutions to simultaneous causality bias:
Randomized controlled experiment
Develop and estimate a complete model of both directions of causality.
Use instrumental variables regression to estimate causal effect of interest – effect of X on Y ignoring the effect of Y on X.
External Validity: refers to the extent to which the findings can be generalized to other settings. Are the findings generalizable to other geographic locations? People with different socio-economic backgrounds? Tools to improve the external validity: selection of the sample. Replications of a given evaluation in times / locations. Can the result be applied to other populations, settings, times etc?