Why We Should Not Be Indifferent to Specification Choices for Difference-in-Differences.

OBJECTIVE To evaluate the effects of specification choices on the accuracy of estimates in difference-in-differences (DID) models. DATA SOURCES Process-of-care quality data from Hospital Compare between 2003 and 2009. STUDY DESIGN We performed a Monte Carlo simulation experiment to estimate the effect of an imaginary policy on quality. The experiment was performed for three different scenarios in which the probability of treatment was (1) unrelated to pre-intervention performance; (2) positively correlated with pre-intervention levels of performance; and (3) positively correlated with pre-intervention trends in performance. We estimated alternative DID models that varied with respect to the choice of data intervals, the comparison group, and the method of obtaining inference. We assessed estimator bias as the mean absolute deviation between estimated program effects and their true value. We evaluated the accuracy of inferences through statistical power and rates of false rejection of the null hypothesis. PRINCIPAL FINDINGS Performance of alternative specifications varied dramatically when the probability of treatment was correlated with pre-intervention levels or trends. In these cases, propensity score matching resulted in much more accurate point estimates. The use of permutation tests resulted in lower false rejection rates for the highly biased estimators, but the use of clustered standard errors resulted in slightly lower false rejection rates for the matching estimators. CONCLUSIONS When treatment and comparison groups differed on pre-intervention levels or trends, our results supported specifications for DID models that include matching for more accurate point estimates and models using clustered standard errors or permutation tests for better inference. Based on our findings, we propose a checklist for DID analysis.

[1]  Daniel P. Kessler,et al.  Impact of malpractice reforms on the supply of physician services. , 2005, JAMA.

[2]  J. Gruber,et al.  How Did Health Care Reform in Massachusetts Impact Insurance Premiums? , 2012, The American economic review.

[3]  J. Reschovsky,et al.  The Effects of SCHIP on Children’s Health Insurance Coverage: Early Evidence from the Community Tracking Study , 2002, Medical care research and review : MCRR.

[4]  M. D. Ernst.  Permutation Methods: A Basis for Exact Inference , 2004 .

[5]  A. Jha,et al.  Association of public reporting for percutaneous coronary intervention with utilization and outcomes among Medicare beneficiaries with acute myocardial infarction. , 2012, JAMA.

[6]  A. Ryan,et al.  The effect of the MassHealth hospital pay-for-performance program on quality. , 2011, Health services research.

[7]  S. Lipsitz,et al.  Massachusetts Health Reform and Disparities in Coverage, Access and Health Status , 2010, Journal of General Internal Medicine.

[8]  Yijuan Chen,et al.  Do healthcare report cards cause providers to select patients and raise quality of care? , 2012, Health economics.

[9]  D. Asch,et al.  Does Exposure to Conflict of Interest Policies in Psychiatry Residency Affect Antidepressant Prescribing? , 2013, Medical care.

[10]  B. Kralj,et al.  Link between Pay for Performance Incentives and Physician Payment Mechanisms: Evidence from the Diabetes Management Incentive in Ontario , 2013, Health economics.

[11]  Stephen G. Donald,et al.  Inference with Difference-in-Differences and Other Panel Data , 2007, The Review of Economics and Statistics.

[12]  Sharon-Lise T. Normand,et al.  Analytic Methods for Constructing Cross-Sectional Profiles of Health Care Providers , 2000, Health Services and Outcomes Research Methodology.

[13]  Rowena Jacobs,et al.  The effect of financial incentives on gatekeeping doctors: evidence from a natural experiment. , 2006, Journal of health economics.

[14]  D. Moher,et al.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement , 2009, BMJ : British Medical Journal.

[15]  B. Sianesi,et al.  PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing, and covariate imbalance testing , 2003 .

[16]  A. Monheit,et al.  Early impact of the Affordable Care Act on health insurance coverage of young adults. , 2012, Health services research.

[17]  Haiden A Huskamp,et al.  Behavioral health insurance parity for federal employees. , 2006, The New England journal of medicine.

[18]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[19]  S. Long,et al.  The impacts of state health reform initiatives on adults in New York and Massachusetts. , 2011, Health services research.

[20]  C. Ai,et al.  Interaction terms in logit and probit models , 2003 .

[21]  B. Starfield,et al.  Impact of State Laws That Extend Eligibility for Parents’ Health Insurance Coverage to Young Adults , 2012, Pediatrics.

[22]  L. Casalino,et al.  Small physician practices in new york needed sustained help to realize gains in quality from use of electronic health records. , 2013, Health affairs.

[23]  D. Polsky,et al.  The effect of pay-for-performance in nursing homes: evidence from state Medicaid programs. , 2013, Health services research.

[24]  H. V. Nguyen.  Do smoke-free car laws work? Evidence from a quasi-experiment. , 2013, Journal of health economics.

[25]  Yu‐Chu Shen,et al.  The effect of Medicaid payment generosity on access and use among beneficiaries. , 2005, Health services research.

[26]  Spencer S Jones,et al.  Electronic health record adoption and quality improvement in US hospitals. , 2010, The American journal of managed care.

[27]  B. Druss,et al.  Change in health insurance coverage in Massachusetts and other New England States by perceived health status: potential impact of health reform. , 2013, American journal of public health.

[28]  N. Dreyer,et al.  The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution , 2014, Journal of managed care & specialty pharmacy.

[29]  T. Breurch,et al.  A simple test for heteroscedasticity and random coefficient variation (econometrica vol 47 , 1979 .

[30]  Jerry A. Hausman,et al.  Attrition Bias in Experimental and Panel Data: The Gary Income Maintenance Experiment , 1979 .

[31]  David M. Drukker,et al.  Testing for Serial Correlation in Linear Panel-data Models , 2003 .

[32]  Kara E MacLeod,et al.  The Use, Safety and Cost of Bariatric Surgery Before and After Medicare's National Coverage Decision , 2011, Annals of surgery.

[33]  Rembert De Blander,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2011 .

[34]  K. Chay,et al.  The Central Role of Noise in Evaluating Interventions that Use Test Scores to Rank Schools , 2003 .

[35]  J. Sekhon,et al.  Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies , 2006, Review of Economics and Statistics.

[36]  Alan Manning,et al.  The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con Out of Econometrics , 2010, SSRN Electronic Journal.

[37]  Joshua D. Angrist,et al.  The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con Out of Econometrics , 2010, SSRN Electronic Journal.

[38]  David C. Chan,et al.  Insurance expansion in Massachusetts did not reduce access among previously insured Medicare patients. , 2013, Health affairs.

[39]  Yulei He,et al.  Health care spending and quality in year 1 of the alternative quality contract. , 2011, The New England journal of medicine.

[40]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[41]  Robin Burgess,et al.  Can Labor Regulation Hinder Economic Performance? Evidence from India , 2004 .

[42]  J. Cappelleri,et al.  Impact of a Pregabalin Step Therapy Policy Among Medicare Advantage Beneficiaries , 2013, Pain Practice.

[43]  Alisa B. Busch,et al.  Impact of Full Mental Health and Substance Abuse Parity for Children in the Federal Employees Health Benefits Program , 2007, Pediatrics.

[44]  M. Roland,et al.  Reduced Mortality With Hospital Pay for Performance in England , 2013 .

[45]  S. Hohmann,et al.  Improved bariatric surgery outcomes for Medicare beneficiaries after implementation of the medicare national coverage determination. , 2010, Archives of surgery.

[46]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[47]  L. Casalino,et al.  Medicare's flagship test of pay-for-performance did not spur more rapid quality improvement among low-performing hospitals. , 2012, Health affairs.

[48]  E John Orav,et al.  The long-term effect of premier pay for performance on patient outcomes. , 2012, The New England journal of medicine.

[49]  T. Siedler,et al.  One Last Puff? Public Smoking Bans and Smoking Behavior , 2010, Journal of health economics.

[50]  S. Soumerai,et al.  Two-year Trends in Cancer Screening Among Low Socioeconomic Status Women in an HMO-based High-deductible Health Plan , 2012, Journal of General Internal Medicine.

[51]  K. Kozhimannil,et al.  The Impact of High-deductible Health Plans on Men and Women: An Analysis of Emergency Department Care , 2013, Medical care.

[52]  B. Fireman,et al.  Effect of influenza vaccination on hospitalizations in persons aged 50 years and older. , 2010, Vaccine.

[53]  C. Hollenbeak,et al.  Financial and Clinical Impact of Team-Based Treatment for Medicaid Enrollees With Diabetes in a Federally Qualified Health Center , 2008, Diabetes Care.

[54]  E. Duflo,et al.  How Much Should We Trust Differences-in-Differences Estimates? , 2001 .

[55]  A. Ryan.  Effects of the Premier Hospital Quality Incentive Demonstration on Medicare patient mortality and cost. , 2009, Health services research.

[56]  A. Ryan,et al.  Medicare's public reporting initiative on hospital quality had modest or no impact on mortality from three key conditions. , 2012, Health affairs.

[57]  Jean M. Mitchell.  Do Financial Incentives Linked to Ownership of Specialty Hospitals Affect Physicians’ Practice Patterns? , 2008, Medical care.

[58]  P. Rosenbaum,et al.  Mortality among patients in VA hospitals in the first 2 years following ACGME resident duty hour reform. , 2007, JAMA.

[59]  David Phelan,et al.  Clinical Effectiveness and Cost Benefit of Universal versus Targeted Methicillin-Resistant Staphylococcus aureus Screening upon Admission in Hospitals , 2011, Infection Control & Hospital Epidemiology.

[60]  T. Pham-Gia,et al.  The mean and median absolute deviations , 2001 .