Comparing Experimental and Matching Methods Using a Large-Scale Voter Mobilization Experiment

In the social sciences, randomized experimentation is the optimal research design for establishing causation. However, for a number of practical reasons, researchers are sometimes unable to conduct experiments and must rely on observational data. In an effort to develop estimators that can approximate experimental results using observational data, scholars have given increasing attention to matching. In this article, we test the performance of matching by gauging the success with which matching approximates experimental results. The voter mobilization experiment presented here comprises a large number of observations (60,000 randomly assigned to the treatment group and nearly two million assigned to the control group) and a rich set of covariates. This study is analyzed in two ways. The first method, instrumental variables estimation, takes advantage of random assignment in order to produce consistent estimates. The second method, matching estimation, ignores random assignment and analyzes the data as though they were nonexperimental. Matching is found to produce biased results in this application because even a rich set of covariates is insufficient to control for preexisting differences between the treatment and control group. Matching, in fact, produces estimates that are no more accurate than those generated by ordinary least squares regression. The experimental findings show that brief paid get-out-the-vote phone calls do not increase turnout, while matching and regression show a large and significant effect.

[1]  D. Rubin.  Matched Sampling for Causal Effects: Matching to Remove Bias in Observational Studies , 1973 .

[2]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[3]  R. Lalonde.  Evaluating the Econometric Evaluations of Training Programs with Experimental Data , 1984 .

[4]  D. Rubin,et al.  The Bias Due to Incomplete Matching , 1985 .

[5]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[6]  Steven J. Rosenstone,et al.  Mobilization, participation, and democracy in America , 1993 .

[7]  Lawrence W. Sherman,et al.  Deterrent effects of police raids on crack houses: A randomized, controlled experiment , 1995 .

[8]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme , 1997 .

[9]  James J. Heckman,et al.  Characterizing Selection Bias Using Experimental Data , 1998 .

[10]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator , 1998 .

[11]  D. Green,et al.  The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment , 2000, American Political Science Review.

[12]  Petra E. Todd,et al.  Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods , 2001 .

[13]  Jeffrey A. Smith,et al.  Does Matching Overcome Lalonde's Critique of Nonexperimental Estimators? , 2000 .

[14]  Howard S. Bloom,et al.  Can Nonexperimental Comparison Group Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs? , 2002 .

[15]  E. Plutzer.  Becoming a Habitual Voter: Inertia, Resources, and Growth in Young Adulthood , 2002, American Political Science Review.

[16]  Sascha O. Becker,et al.  Estimation of Average Treatment Effects Based on Propensity Scores , 2002 .

[17]  Steven Glazerman,et al.  Nonexperimental Versus Experimental Estimates of Earnings Impacts , 2003 .

[18]  Paul E. Peterson,et al.  The Education Gap: Vouchers and Urban Schools , 2003 .

[19]  G. Imbens.  Semiparametric Estimation of Average Treatment Eects under Exogeneity: A Review , 2003 .

[20]  D. Green,et al.  Get Out the Vote!: How to Increase Voter Turnout , 2004 .

[21]  G. Imbens,et al.  Large Sample Properties of Matching Estimators for Average Treatment Effects , 2004 .

[22]  Edward H. Kaplan,et al.  Problems and Methods in the Study of Politics: The illusion of learning from observational research , 2004 .

[23]  Jason Barabas,et al.  How Deliberation Affects Policy Opinions , 2004, American Political Science Review.

[24]  G. Imbens.  Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review , 2004 .

[25]  Do Phone Calls Increase Voter Turnout? An Update , 2005 .

[26]  Kosuke Imai,et al.  Do Get-Out-the-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments , 2005, American Political Science Review.

[27]  Rajeev Dehejia.  Practical propensity score matching: a reply to Smith and Todd , 2005 .

[28]  Jasjeet S. Sekhon,et al.  Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching Package for R , 2008 .