A Politically Robust Experimental Design for Public Policy Evaluation, With Application to the Mexican Universal Health Insurance Program

We develop an approach to conducting large-scale randomized public policy experiments intended to be more robust to the political interventions that have ruined some or all parts of many similar previous efforts. Our proposed design is insulated from selection bias in some circumstances even if we lose observations; our inferences can still be unbiased even if politics disrupts any two of the three steps in our analytical procedures; and other empirical checks are available to validate the overall design. We illustrate with a design and empirical validation of an evaluation of the Mexican Seguro Popular de Salud (Universal Health Insurance)program we are conducting. Seguro Popular, which is intended to grow to provide medical care, drugs, preventative services, and financial health protection to the 50 million Mexicans without health insurance, is one of the largest health reforms of any country in the last two decades. The evaluation is also large scale, constituting one of the largest policy experiments to date and what may be the largest randomized health policy experiment ever.

[1]  Robinson G. Hollister,et al.  How Close Is Close Enough? Evaluating Propensity Score Matching Using Data From A Class Size Reduction Experiment , 2007 .

[2]  H. Bloom,et al.  Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions , 2007 .

[3]  S. Raudenbush,et al.  Strategies for Improving Precision in Group-Randomized Experiments , 2007 .

[4]  Steven Glazerman Daniel P Mayer Paul T Decker Alternative Routes to Teaching: The Impacts of Teach For America on Student Achievement and Other Outcomes , 2006 .

[5]  Felicia Marie Knaul,et al.  Comprehensive reform to improve health system performance in Mexico , 2006, The Lancet.

[6]  D. Greenberg,et al.  Do Experimental and Nonexperimental Evaluations Give Different Answers about the Effectiveness of Government-Funded Training Programs?. , 2006 .

[7]  Gary King,et al.  Zelig: Everyone's Statistical Software , 2006 .

[8]  David W. Nickerson Scalable Protocols Offer Efficient Design for Field Experiments , 2005, Political Analysis.

[9]  Dennis F. Thompson Democracy in Time: Popular Sovereignty and Temporal Representation , 2005 .

[10]  Kosuke Imai,et al.  Do Get-Out-the-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments , 2005, American Political Science Review.

[11]  David H. Greenberg,et al.  Digest of Social Experiments , 2004 .

[12]  G. Harrison,et al.  Field experiments , 1924, The Journal of Agricultural Science.

[13]  T. Dee,et al.  Does merit pay reward good teachers? Evidence from a randomized experiment , 2004 .

[14]  Jeffrey H Silber,et al.  Optimal multivariate matching before randomization. , 2004, Biostatistics.

[15]  William G. Howell Dynamic selection effects in means-tested, urban school voucher programs , 2004 .

[16]  J. Grimshaw,et al.  Estimating the Effects of Interventions That are Deployed in Many Places , 2004 .

[17]  C. Murray,et al.  Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research , 2003, American Political Science Review.

[18]  P. Gertler Do Conditional Cash Transfers Improve Child Health? Evidence from PROGRESA’s Control Randomized Experiment. , 2004, The American economic review.

[19]  F. Knaul,et al.  Evidence-based health policy: three generations of reform in Mexico , 2003, The Lancet.

[20]  F. Ozkan,et al.  Macroeconomic Policies of Developed Democracies. , 2003 .

[21]  Steven Raphael,et al.  Public Transit and the Spatial Distribution of Minority Employment: Evidence from a Natural Experiment , 2003 .

[22]  A. Petrosino,et al.  The “Experimenting Agency” , 2003, Evaluation review.

[23]  M. Killingsworth,et al.  The use of client surveys to gauge the threat of contamination in welfare reform experiments , 2003 .

[24]  D. Rubin,et al.  Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Comments. Authors' reply , 2003 .

[25]  Christopher J L Murray,et al.  Health systems performance assessment: debates, methods and empiricism. , 2003 .

[26]  Xiao-Hua Zhou,et al.  Clustered encouragement designs with individual noncompliance: bayesian inference with randomization, and application to advance directive forms. , 2002, Biostatistics.

[27]  E. Posner,et al.  Legislative Entrenchment: A Reappraisal , 2002 .

[28]  G. King,et al.  Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation , 2001, American Political Science Review.

[29]  Peter J. Bickel,et al.  INFERENCE FOR SEMIPARAMETRIC MODELS: SOME QUESTIONS AND AN ANSWER , 2001 .

[30]  Petra E. Todd,et al.  International Food Policy Research Institute Randomness in the Experimental Samples of Progresa (education, Health, and Nutrition Program) , 2001 .

[31]  Peter Rousseeuw,et al.  Econometric Applications of High-Breakdown Robust Regression Techniques , 2017, 1709.00181.

[32]  Allan Donner,et al.  Design and Analysis of Cluster Randomization Trials in Health Research , 2001 .

[33]  Jason Wittenberg,et al.  Making the Most Of Statistical Analyses: Improving Interpretation and Presentation , 2000 .

[34]  D. Rubin,et al.  Assessing the effect of an influenza vaccine in an encouragement design. , 2000, Biostatistics.

[35]  M. Ruel,et al.  AN OPERATIONS EVALUATION OF PROGRESA FROM THE PERSPECTIVE OF BENEFICIARIES, PORMOTORAS, SCHOOL DIRECTORS AND HEALTH STAFF: FINAL REPORT , 2000 .

[36]  M. Ruel,et al.  AN OPERATIONS EVALUATION OF PROGRESA FROM THE PERSPECTIVE OF BENEFICIARIES, PROMOTORAS, SCHOOL DIRECTORS, AND HEALTH STAFF , 2000 .

[37]  David M. Murray,et al.  Design and Analysis of Group- Randomized Trials , 1998 .

[38]  A Donner,et al.  The merits of matching in community intervention trials: a cautionary tale. , 1997, Statistics in medicine.

[39]  A. Krueger,et al.  Experimental Estimates of Education Production Functions , 1997 .

[40]  Robert F. Boruch,et al.  Randomized experiments for planning and evaluation , 1996 .

[41]  Terry C. Blum,et al.  Assessing the Non-Random Sampling Effects of Subject Attrition in Longitudinal Research , 1996 .

[42]  Bruce Western,et al.  Concepts and Suggestions for Robust Regression Analysis , 1995 .

[43]  James J. Heckman,et al.  Assessing the Case for Social Experiments , 1995 .

[44]  Gary Burtless,et al.  The Case for Randomized Field Trials in Economic and Policy Research , 1995 .

[45]  H. Wainer,et al.  Differential item functioning , 1995 .

[46]  Charles F. Manski,et al.  Evaluating Welfare and Training Programs. , 1994 .

[47]  James J. Heckman,et al.  Randomization and Social Policy Evaluation , 1991 .

[48]  S. Zeger,et al.  On estimating efficacy from clinical trials. , 1991, Statistics in medicine.

[49]  Alberto Alesina,et al.  A Positive Theory of Fiscal Deficits and Government Debt , 1990 .

[50]  森谷 正規,et al.  JOURNAL OF Policy Analysis and Management , 2021, Journal of Policy Analysis and Management.

[51]  M S Kramer,et al.  Scientific challenges in the application of randomized trials. , 1984, JAMA.

[52]  Peter Bohm,et al.  Are there practicable demand-revealing mechanisms? , 1984 .

[53]  B. Flay,et al.  Overcoming Design Problems in Evaluating Health Behavior Programs , 1982, Evaluation & the health professions.

[54]  C. I. Schottland Policymaking for Social Security , 1980 .

[55]  P. Shrout Quasi-experimentation: Design and analysis issues for field settings: by Thomas D. Cook and Donald T. Campbell. Chicago: Rand McNally, 1979 , 1980 .

[56]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[57]  B. Rosner,et al.  Analytic methods in matched pair epidemiological studies. , 1978, International journal of epidemiology.

[58]  Sidney Addelman,et al.  trans-Dimethanolbis(1,1,1-trifluoro-5,5-dimethylhexane-2,4-dionato)zinc(II) , 2008, Acta crystallographica. Section E, Structure reports online.

[59]  Leslie Kish,et al.  A Procedure for Objective Respondent Selection within the Household , 1949 .