Qualitative Comparative Analysis: How Inductive Use and Measurement Error Lead to Problematic Inference

An increasing number of analyses in various subfields of political science employ boolean algebra as proposed in Ragin's (1987) qualitative comparative analysis (QCA). This type of analysis is perfectly justifiable if the goal is to test deterministic hypotheses under the assumption of error-free measures of the employed variables. My contention is, however, that only in a very few research areas are our theories sufficiently advanced to yield deterministic hypotheses. Also, given the nature of our objects of study, error-free measures are largely an illusion. Hence, it is unsurprising that many studies employ QCA inductively and gloss over possible measurement errors. In this paper I address these issues and demonstrate the consequences of these problems with simple empirical examples. In an analysis similar to Monte Carlo simulation I show that using boolean algebra in an exploratory fashion without considering possible measurement errors may lead to dramatically misleading inferences. I then suggest remedies that help researchers to circumvent some of these pitfalls.

[1]  Arild Underdal,et al.  Regime consequences : methodological challenges and research strategies , 2004 .

[2]  Robert O. Keohane,et al.  Designing Social Inquiry: Scientific Inference in Qualitative Research. , 1995 .

[3]  Henry E. Brady,et al.  The Oxford Handbook of Political Methodology , 2010 .

[4]  A. Alesina,et al.  Public Goods and Ethnic Divisions , 1997 .

[5]  Gary Goertz,et al.  The Methodology of Necessary Conditions , 2000 .

[6]  E. Hanushek.  Statistical methods for social scientists , 1977 .

[7]  Jason Seawright.  Testing for Necessary and/or Sufficient Causation: Which Cases Are Relevant? , 2002, Political Analysis.

[8]  Adrian Duşa,et al.  User manual for the QCA(GUI) package in R , 2007 .

[9]  Claudius Wagemann,et al.  Qualitative Comparative Analysis (QCA) und fuzzy sets: ein Lehrbuch für Anwender und jene, die es werden wollen , 2007 .

[10]  Jasjeet S. Sekhon,et al.  Quality Meets Quantity: Case Studies, Conditional Probability, and Counterfactuals , 2004, Perspectives on Politics.

[11]  Benoît Rihoux,et al.  A Commented Review of Applications , 2009 .

[12]  Two Methodological Worlds Apart? Praises and Critiques from a European Comparativist , 2006, Political Analysis.

[13]  Jacqueline H. R. DeMeritt,et al.  The Split Population Logit (SPopLogit): Modeling Measurement Bias in Binary Data , 2011 .

[14]  Benoît Rihoux,et al.  Addressing the critiques of QCA , 2009 .

[15]  Jason Seawright,et al.  Qualitative comparative analysis vis-à-vis regression , 2005 .

[16]  James Mahoney,et al.  Scope in Case-Study Research , 2009 .

[17]  Andrew Bennett.  Stirring the Frequentist Pot with a Dash of Bayes , 2006, Political Analysis.

[18]  Alexander Hicks,et al.  The programmatic emergence of the social security state , 1995 .

[19]  Alexander Hicks,et al.  The comparative political economy of the welfare state , 1995 .

[20]  B. Ebbinghaus,et al.  When Less is More , 2005 .

[21]  S. Hug.  The Effect of Misclassifications in Probit Models: Monte Carlo Simulations and Applications , 2006, Political Analysis.

[22]  Bojana Lobe,et al.  The Case for Qualitative Comparative Analysis (QCA) : Adding Leverage for Thick Cross-Case Comparison , 2010 .

[23]  Philip A. Schrodt.  Beyond the Linear Frequentist Orthodoxy , 2006, Political Analysis.

[24]  B. Rihoux.  Qualitative Comparative Analysis (QCA) and Related Systematic Comparative Methods , 2006 .

[25]  Kevin A. Clarke.  The Reverend and the Ravens: Comment on Seawright , 2002, Political Analysis.

[26]  Jason Seawright.  Assumptions, causal inference, and the goals of QCA , 2005 .

[27]  J. Hausman,et al.  Misclassification of the dependent variable in a discrete-response setting , 1998 .

[28]  Carsten Q. Schneider,et al.  Standards of Good Practice in Qualitative Comparative Analysis (QCA) and Fuzzy-Sets , 2010 .

[29]  K. Bollen,et al.  MACROCOMPARATIVE RESEARCH METHODS , 1993 .

[30]  James Mahoney,et al.  Qualitative Methodology and Comparative Politics , 2007 .

[31]  Bernard Grofman,et al.  An Introduction to Crisp Set QCA, with a Comparison to Binary Logistic Regression , 2009 .

[32]  Charles C. Ragin.  Core versus tangential assumptions in comparative research , 2005 .

[33]  Daniel Little,et al.  Causal Explanation in the Social Sciences , 1996 .

[34]  Charles C. Ragin,et al.  Political Methodology: Qualitative Methods , 1998 .

[35]  Douglas Dion,et al.  Evidence and Inference in the Comparative Case StudyEvidence and Inference in the Comparative Case Study , 1998 .

[36]  How qualitative research really counts , 2005 .

[37]  Henry E. Brady,et al.  Toward a Pluralistic Vision of Methodology , 2006, Political Analysis.

[38]  Benoît Rihoux,et al.  Qualitative Comparative Analysis (QCA) As an Approach , 2009 .

[39]  D. Waldner.  It Ain'T Necessarily So-Or Is It? , 2005 .

[40]  Bear F. Braumoeller.  Causal Complexity and the Study of Politics , 2003, Political Analysis.

[41]  Charles C. Ragin,et al.  Redesigning social inquiry , 2008 .

[42]  Christopher H. Achen.  Two cheers for Charles Ragin , 2005 .

[43]  Carsten Q. Schneider,et al.  Reducing complexity in Qualitative Comparative Analysis (QCA): Remote and proximate factors and the consolidation of democracy , 2006 .

[44]  Stanley Lieberson,et al.  Small N's and Big Conclusions: An Examination of the Reasoning in Comparative Studies Based on a Small Number of Cases , 1991 .

[45]  John Stuart Mill.  A System of Logic, Ratiocinative and Inductive: Of the Four Methods of Experimental Inquiry , 2011 .

[46]  Detlef F. Sprinz,et al.  Models, Numbers, and Cases: Methods for Studying International Relations , 2004 .

[47]  Charles C. Ragin,et al.  The Comparative Political Economy of the Welfare State: A qualitative comparative analysis of pension systems , 1994 .

[48]  M. Kendall,et al.  The Logic of Scientific Discovery. , 1959 .

[49]  J. H. Goldthorpe,et al.  Current Issues in Comparative Macrosociology: A Debate on Methodological Issues , 1997 .

[50]  Jason Seawright.  What Counts as Evidence? Reply , 2002, Political Analysis.

[51]  Daniele Caramani,et al.  Introduction to the Comparative Method With Boolean Algebra , 2008 .

[52]  J. Visser,et al.  When Institutions Matter - Union growth and Decline in Western Europe , 1999 .

[53]  William Roberts Clark,et al.  A Simple Multivariate Test for Asymmetric Hypotheses , 2006, Political Analysis.

[54]  Charles C. Ragin,et al.  Fuzzy-Set Social Science , 2001 .

[55]  Cristina Corduneanu-Huci,et al.  Running Uphill: Political Opportunity in Non-democracies , 2003 .

[56]  R. Goodin,et al.  A new handbook of political science , 1998 .

[57]  Gary Goertz,et al.  A Tale of Two Cultures: Contrasting Quantitative and Qualitative Research , 2006, Political Analysis.

[58]  W. P. Shively.  Case Selection: Insights from Rethinking Social Inquiry , 2006, Political Analysis.

[59]  Paul Pennings.  The Diversity and Causality of Welfare State Reforms Explored with Fuzzy-Sets , 2005 .

[60]  Svend-Erik Skaaning,et al.  Assessing the Robustness of Crisp-set and Fuzzy-set QCA Results , 2011 .

[61]  Olav Schram Stokke,et al.  Boolean Analysis, Mechanisms, and the Study of Regime Effectiveness , 2004 .

[62]  A. Gelman,et al.  Bridges between Deterministic and Probabilistic Models for Binary Data , 2000 .

[63]  Charles C. Ragin,et al.  Set Relations in Social Research: Evaluating Their Consistency and Coverage , 2006, Political Analysis.

[64]  Bear F. Braumoeller,et al.  Watching Your Posterior: Comment on Seawright , 2002, Political Analysis.

[65]  David-Hillel Ruben,et al.  Varieties Of Social Explanation: An Introduction To The Philosophy Of Social Science , 1991 .

[66]  M. Maggetti,et al.  Dealing with Errors in QCA , 2013 .