New Evidence on the Finite Sample Properties of Propensity Score Reweighting and Matching Estimators

Abstract Frölich (2004) compares the finite sample properties of reweighting and matching estimators of average treatment effects and concludes that reweighting performs far worse than even the simplest matching estimator. We argue that this conclusion is unjustified. Neither approach dominates the other uniformly across data-generating processes (DGPs). Expanding on Frölich's analysis, this paper analyzes empirical as well as hypothetical DGPs and also examines the effect of misspecification. We conclude that reweighting is competitive with the most effective matching estimators when overlap is good, but that matching may be more effective when overlap is sufficiently poor.

[1]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[2]  D. Rubin.  Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[3]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[4]  Hedley Rees,et al.  Limited-Dependent and Qualitative Variables in Econometrics. , 1985 .

[5]  James J. Heckman,et al.  Alternative methods for evaluating the impact of interventions: An overview , 1985 .

[6]  J. Heckman,et al.  Longitudinal Analysis of Labor Market Data: Alternative methods for evaluating the impact of interventions , 1985 .

[7]  P. Rosenbaum.  Model-Based Direct Adjustment , 1987 .

[8]  C. Drake.  Effects of misspecification of the propensity score on estimators of treatment effect , 1993 .

[9]  J. Robins,et al.  Estimation of Regression Coefficients When Some Regressors are not Always Observed , 1994 .

[10]  Theo Gasser,et al.  Finite-Sample Variance of Local Polynomials: Analysis and Solutions , 1996 .

[11]  J. Dinardo,et al.  Labor Market Institutions and the Distribution of Wages, 1973-1992: A Semiparametric Approach , 1996 .

[12]  J. Hahn.  On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects , 1998 .

[13]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator , 1998 .

[14]  Rebecca M. Blank,et al.  Race and gender in the labor market , 1999 .

[15]  Theo Gasser,et al.  Data Adaptive Ridging in Local Polynomial Regression , 2000 .

[16]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[17]  Jeffrey M. Wooldridge,et al.  Solutions Manual and Supplementary Materials for Econometric Analysis of Cross Section and Panel Data , 2003 .

[18]  Jeffrey A. Smith,et al.  Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators? , 2000 .

[19]  Joseph P. Lupton,et al.  Accounting for the Black–White Wealth Gap , 2001 .

[20]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[21]  J. Wooldridge.  Inverse probability weighted estimation for general missing data problems , 2004 .

[22]  J. Lunceford,et al.  Stratification and weighting via the propensity score in estimation of causal treatment effects: a comparative study , 2004, Statistics in medicine.

[23]  G. Imbens,et al.  Large Sample Properties of Matching Estimators for Average Treatment Effects , 2004 .

[24]  Markus Frlich,et al.  Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators , 2004, Review of Economics and Statistics.

[25]  D. Black,et al.  How robust is the evidence on the effects of college quality? Evidence from matching , 2004 .

[26]  G. Imbens,et al.  Implementing Matching Estimators for Average Treatment Effects in Stata , 2004 .

[27]  Joseph Kang,et al.  Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data , 2007, 0804.2958.

[28]  G. Imbens,et al.  On the Failure of the Bootstrap for Matching Estimators , 2006 .

[29]  Thomas Lemieux,et al.  Increasing Residual Wage Inequality: Composition Effects, Noisy Data, or Rising Demand for Skill? , 2006 .

[30]  Marie Davidian,et al.  Comment: Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data. , 2008, Statistical science : a review journal of the Institute of Mathematical Statistics.

[31]  Jeffrey A. Smith,et al.  Bandwidth Selection and the Estimation of Treatment Effects with Unbalanced Data , 2007, SSRN Electronic Journal.

[32]  J. Robins,et al.  Comment: Performance of Double-Robust Estimators When “Inverse Probability” Weights Are Highly Variable , 2007, 0804.2965.

[33]  G. Imbens,et al.  Bias-Corrected Matching Estimators for Average Treatment Effects , 2002 .

[34]  Guido W. Imbens,et al.  Estimation of the Conditional Variance in Paired Experiments , 2008 .

[35]  B. Graham,et al.  Inverse Probability Tilting for Moment Condition Models with Missing Data , 2008 .

[36]  Optimal Bandwidth Choice for Matching Estimators by Double Smoothing , 2008 .

[37]  Anton Flossmann.  Empirical Bias Bandwidth Choice for Local Polynomial Matching Estimators , 2008 .

[38]  Richard K. Crump,et al.  Dealing with limited overlap in estimation of average treatment effects , 2009 .

[39]  Zhiqiang Tan,et al.  Bounded, efficient and doubly robust estimation with inverse weighting , 2010 .

[40]  M. Lechner,et al.  How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score , 2010, SSRN Electronic Journal.

[41]  Shakeeb Khan,et al.  Irregular Identification, Support Conditions, and Inverse Weight Estimation , 2010 .

[42]  Michael Lechner,et al.  The performance of estimators based on the propensity score , 2013 .

[43]  Jooyoung Park,et al.  The impact of import-related displacement on local business activity , 2014 .

[44]  Z. E. Gevrek,et al.  Semiparametric Decomposition of the Gender Achievement Gap : An Application for Turkey , 2014 .

[45]  Chris Muris,et al.  Model averaging in semiparametric estimation of treatment effects , 2015 .

[46]  M. Leibbrandt,et al.  Health Outcomes for Children Born to Teen Mothers in Cape Town, South Africa , 2015, Economic Development and Cultural Change.

[47]  Ling Lei Lisic,et al.  Executive Overconfidence and Compensation Structure , 2015 .

[48]  Tyler A. Scott.  Does Collaboration Make Any Difference? Linking Collaborative Governance to Environmental Outcomes. , 2015, Journal of policy analysis and management : [the journal of the Association for Public Policy Analysis and Management].

[49]  G. Bush,et al.  The effects of location-based tax policies on the distribution of household income: Evidence from the federal Empowerment Zone program , 2015 .

[50]  Joel Stiebale,et al.  Cross-Border M&As and Innovative Activity of Acquiring and Target Firms , 2016 .

[51]  I. Idris,et al.  Comparative Efficacy of Adding Sitagliptin to Metformin, Sulfonylurea or Dual Therapy: A Propensity Score-Weighted Cohort Study , 2015, Diabetes Therapy.

[52]  Nandita Mitra,et al.  Propensity score and doubly robust methods for estimating the effect of treatment on censored cost , 2016, Statistics in medicine.

[53]  E. Heinesen,et al.  Labour market participation after breast cancer for employees from the private and public sectors: Educational and sector gradients in the effect of cancer. , 2016, Economics and human biology.

[54]  A. Maffioli,et al.  Extension and Matching Grants for Improved Management: An Evaluation of the Uruguayan Livestock Program , 2016 .