CHAPTER 5 A Typology of Strategies for Ruling Out Threats to Validity

Don Campbell’s name will long be associated with the notion of threats to validity. In part, this is because he provided memorable and pedagogically appealing labels for, and categorizations of, threats to validity (Campbell, 1957; Campbell & Stanley, 1966; Cook & Campbell, 1979). Equally consequential have been Campbell’s demonstrations of the pernicious nature and omnipresence of threats to validity (e.g., Campbell & Boruch, 1975; Campbell & Erlebacher, 1970) and his explication of an epistemology whereby the effects of a treatment are assessed by ruling out plausible rival hypotheses (Campbell, 1974, 1988). The genius of Campbell’s (1969) contribution is also manifest in his imaginative illustrations of methods for ruling out threats to validity. The present chapter, which I hope is faithful to the Campbellian tradition, provides a typology of strategies by which threats to validity can be ruled out. Researchers have long been ruling out threats to validity using a wide variety of methods. The typology I propose explicates the logic by which all these methods work. By understanding the logic behind their methods, researchers should be able to rule out threats to validity more effectively and, equally important, be better able to assess howeffectively they have ruled them out.

[1]  P. Meehl.  Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. , 1978 .

[2]  Benton J. Underwood,et al.  Individual differences as a crucible in theory construction. , 1975 .

[3]  D. Phillips,et al.  SAME TIME, LAST YEAR: SELECTIVE DATA DREDGING FOR NEGATIVE FINDINGS* , 1985 .

[4]  Donald T. Campbell,et al.  Determining the Social Effects of a Legal Reform , 1970 .

[5]  David S. Cordray,et al.  Quasi‐experimental analysis: A mixture of methods and judgment , 1986 .

[6]  D T Campbell,et al.  Methods Factors In Multitrait-Multimethod Matrices : Multiplicative Rather Than Additive? , 1967, Multivariate behavioral research.

[7]  Melvin M. Mark,et al.  From program theory to tests of program theory , 1990 .

[8]  C. S. Reichardt,et al.  The Criteria for Convergent and Discriminant Validity in a Multitrait-Multimethod Matrix. , 1995, Multivariate behavioral research.

[9]  J. Minton.  THE IMPACT OF "SESAME STREET" ON READING READINESS OF KINDERGARTEN CHILDREN. , 1972 .

[10]  William M. K. Trochim,et al.  Pattern Matching, Validity, and Conceptualization in Program Evaluation , 1985 .

[11]  Charles S. Reichardt,et al.  Satisfying the constraints of causal modeling , 1986 .

[12]  S. Leibowitz,et al.  Impact of a galanin antagonist on exogenous galanin and natural patterns of fat ingestion , 1992, Brain Research.

[13]  G. Ferro-Luzzi.  On Evolutionary Epistemology , 1982, Current Anthropology.

[14]  Timothy D. Wilson,et al.  Experimentation in social psychology. , 1998 .

[15]  H. F. Gollob,et al.  Ruling Out Threats To Validity , 1989 .

[16]  R. Boruch,et al.  3 – Making the Case for Randomized Assignment to Treatments by Considering the Alternatives: Six Ways in Which Quasi-Experimental Evaluations In Compensatory Education Tend to Underestimate Effects , 1975 .

[17]  D. Phillips,et al.  Motor vehicle fatalities increase just after publicized suicide stories. , 1977, Science.

[18]  R. Horwitz.  The planning of observational studies of human populations , 1979 .

[19]  N. Miller,et al.  Experiments on motivation. Studies combining psychological, physiological, and pharmacological techniques. , 1957, Science.

[20]  F. Lord.  A paradox in the interpretation of group comparisons. , 1967, Psychological bulletin.

[21]  W. G. Cochran.  The Planning of Observational Studies of Human Populations , 1965 .

[22]  R. Zajonc,et al.  Birth Order and Intellectual Development , 1975 .

[23]  P. Meehl.  Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[24]  Charles S. Reichardt,et al.  Taking uncertainty into account when estimating effects , 1987 .

[25]  W. Trochim.  Outcome pattern matching and program theory , 1989 .

[26]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[27]  W. G. Cochran,et al.  Controlling Bias in Observational Studies: A Review. , 1974 .

[28]  Melvin M. Mark,et al.  Validity typologies and the logic and practice of quasi‐experimentation , 1986 .