Matching and Regression to the Mean in Difference‐in‐Differences Analysis

OBJECTIVE To demonstrate regression to the mean bias introduced by matching on preperiod variables in difference-in-differences studies. DATA SOURCES Simulated data. STUDY DESIGN We performed a Monte Carlo simulation to estimate the effect of a placebo intervention on simulated longitudinal data for units in treatment and control groups using unmatched and matched difference-in-differences analyses. We varied the preperiod level and trend differences between the treatment and control groups, and the serial correlation of the matching variables. We assessed estimator bias as the mean absolute deviation of estimated program effects from the true value of zero. PRINCIPAL FINDINGS When preperiod outcome level is correlated with treatment assignment, an unmatched analysis is unbiased, but matching units on preperiod outcome levels produces biased estimates. The bias increases with greater preperiod level differences and weaker serial correlation in the outcome. This problem extends to matching on preperiod level of a time-varying covariate. When treatment assignment is correlated with preperiod trend only, the unmatched analysis is biased, and matching units on preperiod level or trend does not introduce additional bias. CONCLUSIONS Researchers should be aware of the threat of regression to the mean when constructing matched samples for difference-in-differences. We provide guidance on when to incorporate matching in this study design.

[1]  S. Mattke,et al.  Medicare Home Visit Program Associated With Fewer Hospital And Nursing Home Admissions, Increased Office Visits. , 2015, Health affairs.

[2]  Elizabeth A. Stuart,et al.  Using propensity scores in difference-in-differences models to estimate the effects of a policy change , 2014, Health Services and Outcomes Research Methodology.

[3]  A. Jha,et al.  Hospital closures had no measurable impact on local hospitalization rates or mortality rates, 2003-11. , 2015, Health affairs.

[4]  Randall S. Brown,et al.  Two-Year Costs and Quality in the Comprehensive Primary Care Initiative. , 2016, The New England journal of medicine.

[5]  Donald B. Rubin,et al.  Measurement Error and Regression to the Mean in Matched Samples , 1971 .

[6]  Adrian G Barnett,et al.  Regression to the mean: what it is and how to deal with it. , 2004, International journal of epidemiology.

[7]  Q. Mcnemar,et al.  Sampling in psychological research. , 1940 .

[8]  C. Hollenbeak,et al.  Financial and Clinical Impact of Team-Based Treatment for Medicaid Enrollees With Diabetes in a Federally Qualified Health Center , 2008, Diabetes Care.

[9]  Andrew M Ryan,et al.  Why We Should Not Be Indifferent to Specification Choices for Difference-in-Differences. , 2015, Health services research.

[10]  Laurence C Baker,et al.  Integrated telehealth and care management program for Medicare beneficiaries with chronic disease linked to savings. , 2011, Health affairs.

[11]  Jay Bhattacharya,et al.  Do Instrumental Variables Belong in Propensity Scores? , 2007 .

[12]  Cyril F. Chang,et al.  Impact of high-deductible health plans on health care utilization and costs. , 2011, Health services research.

[13]  Soeren Mattke,et al.  Managing manifest diseases, but not health risks, saved PepsiCo money over seven years. , 2014, Health affairs.

[14]  Susan C. Miller,et al.  Changes in Medicare costs with the growth of hospice care in nursing homes. , 2015, The New England journal of medicine.

[15]  M. Rosenthal,et al.  A Difference-in-Difference Analysis of Changes in Quality, Utilization and Cost Following the Colorado Multi-Payer Patient-Centered Medical Home Pilot , 2016, Journal of General Internal Medicine.

[16]  K. Kleinman,et al.  Emergency department use and subsequent hospitalizations among members of a high-deductible health plan. , 2007, JAMA.

[17]  A. Ryan,et al.  Association of Hospital Participation in a Quality Reporting Program with Surgical Outcomes and Expenditures for Medicare Beneficiaries , 2015 .

[18]  J. Avorn,et al.  Variable selection for propensity score models. , 2006, American journal of epidemiology.

[19]  Elizabeth A Stuart,et al.  Matching methods for causal inference: A review and a look forward. , 2010, Statistical science : a review journal of the Institute of Mathematical Statistics.

[20]  Joshua D. Angrist,et al.  Mostly Harmless Econometrics: An Empiricist's Companion , 2008 .

[21]  David Mancuso,et al.  Care coordination program for Washington State Medicaid enrollees reduced inpatient hospital costs. , 2015, Health affairs.

[22]  Arnold Milstein,et al.  Reductions in Mortality Associated With Intensive Public Reporting of Hospital Outcomes , 2008, American journal of medical quality : the official journal of the American College of Medical Quality.

[23]  Andrew M Ryan,et al.  Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries. , 2015, JAMA.