Quality Meets Quantity: Case Studies, Conditional Probability, and Counterfactuals

In contrast to statistical methods, a number of case study methods—collectively referred to as Mill's methods, used by generations of social science researchers—only consider deterministic relationships. They do so to their detriment because heeding the basic lessons of statistical inference can prevent serious inferential errors. Of particular importance is the use of conditional probabilities to compare relevant counterfactuals. A prominent example of work using Mill's methods is Theda Skocpol's States and Social Revolutions. Barbara Geddes's widely assigned critique of Skocpol's claim of a causal relationship between foreign threat and social revolution is valid if this relationship is considered to be deterministic. If, however, we interpret Skocpol's hypothesized causal relationship to be probabilistic, Geddes's data support Skocpol's hypothesis. But Skocpol, unlike Geddes, failed to provide the data necessary to compare conditional probabilities. Also problematic for Skocpol is the fact that when one makes causal inferences, conditional probabilities are of interest only insofar as they provide information about relevant counterfactuals.Jasjeet S. Sekhon thanks Walter R. Mebane Jr., Henry Brady, Bear Braumoeller, Shigeo Hirano, Gary King, John Londregan, Bruce Rusk, Theda Skocpol, Suzanne M. Smith, Jonathan N. Wand, the editors of Perspectives on Politics, and three anonymous reviewers for valuable comments and advice.

[1]  D. Collier Translating Quantitative Methods for Qualitative Researchers: The Case of Selection Bias , 1995, American Political Science Review.

[2]  David Faure,et al.  Revolution and Rebellion in the Early Modern World. By Jack A. Goldstone. University of California Press, 1991. 632 pp. $34.95 , 1991 .

[3]  Jason Seawright Testing for Necessary and/or Sufficient Causation: Which Cases Are Relevant? , 2002, Political Analysis.

[4]  Jasjeet S. Sekhon,et al.  Overvoting and representation: an examination of overvoted presidential ballots in Broward and Miami-Dade counties , 2003 .

[5]  A. Dawid Causal Inference without Counterfactuals , 2000 .

[6]  Stanley Lieberson,et al.  Small N's and Big Conclusions: An Examination of the Reasoning in Comparative Studies Based on a Small Number of Cases , 1991 .

[7]  D. Green,et al.  The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment , 2000, American Political Science Review.

[8]  Theodore W. Frick,et al.  Artificial Tutoring Systems: What Computers Can and Can't Know , 1997 .

[9]  Ernest Nagel,et al.  An Introduction to Logic and Scientific Method , 1934, Nature.

[10]  Larry M. Bartels Pooling Disparate Observations , 1996 .

[11]  Charles C. Ragin,et al.  Political Methodology: Qualitative Methods , 1998 .

[12]  J. Mill A system of logic, ratiocinative and inductive : being a connected view of the principles of evidence, and the methods of scientific investigation. , 2011 .

[13]  J.J.H. Bartholomée,et al.  Campell, Donald T. , 2005 .

[14]  T. Skocpol Vision and Method in Historical Sociology: Emerging Agendas and Recurrent Strategies in Historical Sociology , 1984 .

[15]  John K. Ryan,et al.  An Introduction to Logic and Scientific Method , 1935 .

[16]  Humphrey Thomas Pledge Science since 1500 : a short history of mathematics, physics, chemistry, biology , 1967 .

[17]  Gary Goertz,et al.  The Methodology of Necessary Conditions , 2000 .

[18]  Jack Andrew Goldstone,et al.  Methodological Issues in Comparative Macrosociology , 1997 .

[19]  Bear F. Braumoeller,et al.  Watching Your Posterior: Comment on Seawright , 2002, Political Analysis.

[20]  Jasjeet S. Sekhon,et al.  Black Candidates and Black Voters: Assessing the Impact of Candidate Race on Uncounted Vote Rates , 2005 .

[21]  Charles C. Ragin,et al.  Fuzzy-Set Social Science , 2001 .

[22]  Roland P. Falkner,et al.  History of statistics , 1891 .

[23]  S. Stigler,et al.  The History of Statistics: The Measurement of Uncertainty before 1900 by Stephen M. Stigler (review) , 1986, Technology and Culture.

[24]  J. Goldstone,et al.  Revolution and Rebellion in the Early Modern World. By Jack A. Goldstone. University of California Press, 1991. 632 pp. $34.95 , 1991 .

[25]  M. Burawoy Two methods in search of science , 1989 .

[26]  A. B. Sørensen,et al.  Microfoundations, Method, and Causation: On the Philosophy of the Social Sciences , 1999 .

[27]  J. Vandenbroucke In Defense of Case Reports and Case Series , 2001, Annals of Internal Medicine.

[28]  T. McKeown Case Studies and the Statistical Worldview: Review of King, Keohane, and Verba's Designing Social Inquiry: Scientific Inference in Qualitative Research , 1999, International Organization.

[29]  Stephen M. Stigler,et al.  The History of Statistics: The Measurement of Uncertainty before 1900 , 1986 .

[30]  T. Skocpol,et al.  States and social revolutions : a comparative analysis of France, Russia, and China , 1979 .

[31]  Kevin A. Clarke The Reverend and the Ravens: Comment on Seawright , 2002, Political Analysis.

[32]  Donald B. Rubin,et al.  Principal Stratification Approach to Broken Randomized Experiments , 2003 .

[33]  B. Geddes How the Cases You Choose Affect the Answers You Get: Selection Bias in Comparative Politics , 1990, Political Analysis.

[34]  Douglas Dion,et al.  Evidence and Inference in the Comparative Case StudyEvidence and Inference in the Comparative Case Study , 1998 .

[35]  R. Goodin,et al.  A new handbook of political science , 1998 .

[36]  T. Speed,et al.  On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 , 1990 .

[37]  Jason Seawright What Counts as Evidence? Reply , 2002, Political Analysis.

[38]  Kosuke Imai,et al.  Do Get-Out-the-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments , 2005, American Political Science Review.

[39]  J. Mill A System of Logic , 1843 .

[40]  D. Collier,et al.  Insights and Pitfalls: Selection Bias in Qualitative Research , 1996, World Politics.

[41]  Stanley Lieberson,et al.  More on the Uneasy Case for Using Mill-Type Methods in Small-N Comparative Studies , 1994 .

[42]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[43]  J. Walkup Fuzzy-set social science , 2003 .

[44]  P. Holland Statistics and Causal Inference , 1985 .

[45]  RONALD A. FISHER,et al.  Cancer and Smoking , 1958, Nature.

[46]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[47]  A. Przeworski,et al.  The logic of comparative social inquiry , 1970 .

[48]  T. Skocpol,et al.  States and Social Revolutions: A Comparative Analysis of France, Russia, and China. , 1980 .

[49]  J. Mahoney Nominal, Ordinal, and Narrative Appraisal in Macrocausal Analysis , 1999, American Journal of Sociology.

[50]  Dwight B. Billings States and Social Revolutions: A Comparative Analysis of France, Russia, and China.By Theda Skocpol. Cambridge: Cambridge University Press, 1979. 407 pp. Cloth, $30.00; paper, $7.95 , 1980 .

[51]  A. P. Dawid,et al.  Causal inference without counterfactuals (with Discussion) , 2000 .

[52]  D. Rubin [On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies , 1990 .

[53]  W. S. Robinson The Logical Structure of Analytic Induction , 1951 .

[54]  Jasjeet S. Sekhon,et al.  Robust Estimation and Outlier Detection for Overdispersed Multinomial Models of Count Data , 2004 .

[55]  W. Salmon Four decades of scientific explanation , 1989 .

[56]  RONALD A. FISHER,et al.  Lung Cancer and Cigarettes? , 1958, Nature.

[57]  N. Polsby,et al.  Case Study and Theory in Political Science , 1975 .