QCA is of questionable value for policy research

Abstract Qualitative Comparative Analysis (QCA) has been championed as a valuable tool for public policy research. Focusing on the field of policy evaluation, this research note assesses QCA by comparing research that uses this method to studies based on standard practices for quantitative policy analysis. While attention is centrally focused on causal inference, questions of measurement are also addressed. The analysis suggests that QCA adds little value to current methods of policy scholarship, and its contribution in fact falls far short, compared with present-day standard practices. For example, a properly defined “net effects” framework — which is pointedly rejected by QCA — provides valuable insights regarding the causal effects that are a central concern of policy evaluation. By contrast, as an approach to policy analysis, QCA suffers from severe limitations in both its framework and its findings.

[1]  English Emmenegger Patrick Dunning, Thad (2012): Natural Experiments in the Social Sciences: A Design­Based Approach. Cambridge: , 2012 .

[2]  Carsten Q. Schneider,et al.  Set-Theoretic Methods for the Social Sciences: Potential pitfalls and suggestions for solutions , 2012 .

[3]  Carsten Q. Schneider,et al.  Set-Theoretic Methods for the Social Sciences: Contents , 2012 .

[4]  N. Nicosia,et al.  Junk Food in Schools and Childhood Obesity. , 2012, Journal of policy analysis and management : [the journal of the Association for Public Policy Analysis and Management].

[5]  G. Allison Emergence of Schools of Public Policy: Reflections by a Founding Dean , 2008 .

[6]  E. Hanushek,et al.  The Failure of Input-Based Schooling Policies , 2002 .

[7]  Stephanie Hatheway,et al.  Class and Schools - Using Social, Economic, and Educational Reform to Close the Black-White Achievement Gap , 2006 .

[8]  John Hudson,et al.  Qualitative comparative analysis and applied public policy analysis: New applications of innovative methods , 2013 .

[9]  David Card,et al.  Minimum Wages and Employment: A Case Study of the Fast Food Industry in New Jersey and Pennsylvania , 1993 .

[10]  Jeffrey M. Woodbridge Econometric Analysis of Cross Section and Panel Data , 2002 .

[11]  Sean F. Reardon,et al.  Brown Fades: The End of Court‐Ordered School Desegregation and the Resegregation of American Public Schools , 2012 .

[12]  R. Lalonde Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[13]  Helen F. Ladd,et al.  Status versus Growth: The Distributional Effects of School Accountability Policies. Working Paper. , 2009 .

[14]  Clare Bambra,et al.  Applying Qualitative Comparative Analysis (QCA) to evaluate a public health policy initiative in the North East of England , 2013 .

[15]  E. Hanushek The Economic Value of Higher Teacher Quality , 2010 .

[16]  Paul Pennings The Diversity and Causality of Welfare State Reforms Explored with Fuzzy-Sets , 2005 .

[17]  Andreas Bal T Hasar The Effects of Institutional Design on the Utilization of Evaluation Evidenced Using Qualitative Comparative Analysis (QCA) , 2006 .

[18]  Carsten Q. Schneider,et al.  Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis , 2012 .

[19]  Parag A. Pathak,et al.  Who Benefits from Kipp? , 2012, SSRN Electronic Journal.

[20]  B. Sen Is There an Association Between Gasoline Prices and Physical Activity? Evidence from American Time Use Data , 2012 .

[21]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[22]  Benoît Rihoux,et al.  Qualitative Comparative Analysis (QCA) in Public Policy Analysis : An Extensive Review , 2011 .

[23]  D. Rubin Causal Inference Using Potential Outcomes , 2005 .

[24]  Thad Dunning,et al.  Natural Experiments in the Social Sciences , 2012 .

[25]  Barbara Befani,et al.  QCA as a Tool for Realistic Evaluations. The Case of the Swiss Environmental Impact Assessment , 2006 .

[26]  M. Sobel,et al.  Identification Problems in the Social Sciences. , 1996 .

[27]  R. Johnson Long-Run Impacts of School Desegregation & School Quality on Adult Attainments , 2011 .

[28]  Michael Wooldridge,et al.  Econometric Analysis of Cross Section and Panel Data, 2nd Edition , 2001 .

[29]  Sandra Lowe,et al.  Redesigning Social Inquiry Fuzzy Sets And Beyond , 2016 .

[30]  S. Lee High non-standard employment rates in the Republic of Korea and Japan: Analyzing policy configurations with fuzzy-set/QCA , 2013 .

[31]  Benoît Rihoux,et al.  Innovative Comparative Methods for policy Analysis. Beyond the Quantitative-Qualitative Divide , 2006 .

[32]  David Collier,et al.  Rethinking Social Inquiry: Diverse Tools, Shared Standards , 2004 .

[33]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme , 1997 .

[34]  Joshua D. Angrist,et al.  The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con Out of Econometrics , 2010, SSRN Electronic Journal.

[35]  Jon Stiles,et al.  The Impact of California's Cal-Learn Demonstration Project, Final Report , 2000 .

[36]  Charles C. Ragin,et al.  The Limitations of Net-Effects Thinking , 2006 .

[37]  H. Bloom Learning more from social experiments: evolving analytic approaches , 2006 .