The Replication Problem in Sociology: A Report and a Suggestion*

The deleterious effects of joint bias in favor of statistical inference and against replication are becoming well known. The acceptance of numerous Type I errors into the literature is by far the most serious of these. Data on the contents of three major journals support the contention that a joint bias for statistical significance tests, for rejections, and against replication exists in modern sociology. This finding replicates that of Sterling (1959) for psychology. A speculative analysis of the dynamics of publication decisions suggests that a compact format for reporting replications might make their publication more attractive to editors, and thus increase their frequency in the literature. A possible format for briefly reporting replication studies is suggested.

[1]  S. Labovitz.  The Nonutility of Significance Tests: The Significance of Tests of Significance Reconsidered , 1970 .

[2]  G. William Walster,et al.  A Proposal for a New Editorial Policy in the Social Sciences , 1970 .

[3]  On Replication: Notes on the Notions of a Replicability Quotient and a Generalizability Quotient , 1970 .

[4]  D. Lykken.  Statistical significance in psychological research. , 1968, Psychological bulletin.

[5]  J. Williams,et al.  INTERVIEWER ROLE PERFORMANCE: A FURTHER NOTE ON BIAS IN THE INFORMATION INTERVIEW , 1968 .

[6]  VALIDITY OF QUESTIONNAIRE DATA: REPORTED AND OBSERVED ATTENDANCE IN AN ADULT EDUCATION PROGRAM , 1968 .

[7]  P. Meehl.  Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[8]  D. Bakan,et al.  The test of significance in psychological research. , 1966, Psychological bulletin.

[9]  Alan Stuart,et al.  Data-Dredging Procedures in Survey Analysis , 1966 .

[10]  M. J. Rosenberg,et al.  WHEN DISSONANCE FAILS: ON ELIMINATING EVALUATION APPREHENSION FROM ATTITUDE MEASUREMENT. , 1965, Journal of personality and social psychology.

[11]  M. Orne,et al.  THE CONTRIBUTION OF NONDEPRIVATION FACTORS IN THE PRODUCTION OF SENSORY DEPRIVATION EFFECTS: THE PSYCHOLOGY OF THE "PANIC BUTTON". , 1964, Journal of abnormal psychology.

[12]  M. Orne.  On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[13]  H. Blalock.  Theory, Measurement, and Replication in the Social Sciences , 1961, American Journal of Sociology.

[14]  Quinn McNemar,et al.  At random: Sense and nonsense. , 1960 .

[15]  Gordon Tullock.  Publication Decisions and Tests of Significance—A Comment , 1959 .

[16]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[17]  Leslie Kish,et al.  Some Statistical Problems in Research Design , 1959 .