Testing Attrition Bias in Field Experiments

We approach attrition in field experiments with baseline outcome data as an identification problem in a panel model. A systematic review of the literature indicates that there is no consensus on how to test for attrition bias. We establish identifying assumptions for treatment effects for both the respondent subpopulation and the study population. We then derive their sharp testable implications on the baseline outcome distribution and propose randomization procedures to test them. We demonstrate that the most commonly used test does not control size in general when internal validity holds. Simulations and applications illustrate the empirical relevance of our analysis.

[1]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[2]  David Brownstone,et al.  Multiple Imputation Methodology for Missing Data, Non-Random Response, and Panel Attrition , 1997 .

[3]  D. Altman.  Comparability of Randomised Groups , 1985 .

[4]  G. Imbens,et al.  Exact p-Values for Network Interference , 2015, 1506.02084.

[5]  G. Barrett,et al.  A Comparison of Micro and Macro Expenditure Measures Across Countries Using Differing Survey Methods , 2013 .

[6]  H. White,et al.  Nonparametric Identification in Nonseparable Panel Data Models with Generalized Fixed Effects , 2009 .

[7]  Jeffrey M. Wooldridge,et al.  Selection corrections for panel data models under conditional mean independence assumptions , 1995 .

[8]  Carlos A. Flores,et al.  Bounds on Treatment Effects in the Presence of Sample Selection and Noncompliance: The Wage Effects of Job Corps , 2015 .

[9]  Patrick M. Kline,et al.  Sensitivity to Missing Data Assumptions: Theory and an Evaluation of the U.S. Wage Structure , 2010 .

[10]  J. Angrist,et al.  Identification and Estimation of Local Average Treatment Effects , 1995 .

[11]  Bruce D. Meyer,et al.  Household Surveys in Crisis , 2015 .

[12]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[13]  C. D. Chaisemartin.  Tolerating defiance? Local average treatment effects without monotonicity , 2017 .

[14]  Matthew M. Chingos,et al.  The Review of Economics and Statistics , 2018 .

[15]  Alwyn Young.  Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results* , 2018, The Quarterly Journal of Economics.

[16]  Nicholas L. Wilson,et al.  Is Sex Like Driving? HIV Prevention and Risk Compensation. , 2014, Journal of development economics.

[17]  T. Kitagawa.  A Test for Instrument Validity , 2015 .

[18]  Nonstandard Asymptotics,et al.  Monte Carlo Tests with Nuisance Parameters: A General Approach to Finite-Sample Inference and , 2005 .

[19]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[20]  Isaiah Andrews,et al.  A Simple Approximation for Evaluating External Validity Bias , 2017, Economics Letters.

[21]  J. Angrist.  Conditional Independence in Sample Selection Models , 1997 .

[22]  J. Powell,et al.  Semiparametric estimation of censored selection models with a nonparametric selection mechanism , 1993 .

[23]  Luc Behaghel,et al.  Please Call Again: Correcting Nonresponse Bias in Treatment Effect Models , 2015, Review of Economics and Statistics.

[24]  Rachel Glennerster,et al.  Running Randomized Evaluations: A Practical Guide , 2013 .

[25]  Yuanyuan Wan,et al.  Testing Local Average Treatment Effect Assumptions , 2015, Review of Economics and Statistics.

[26]  K. Macours,et al.  Attrition in Randomized Control Trials: Using Tracking Information to Correct Bias , 2017, SSRN Electronic Journal.

[27]  Jerry A. Hausman,et al.  Attrition Bias in Experimental and Panel Data: The Gary Income Maintenance Experiment , 1979 .

[28]  Xiaoxia Shi,et al.  TESTING GENERALIZED REGRESSION MONOTONICITY , 2019, Econometric Theory.

[29]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[30]  Martin Huber,et al.  Identification of Average Treatment Effects in Social Experiments Under Alternative Forms of Attrition , 2012 .

[31]  Charles F. Manski,et al.  Partial identification with missing data: concepts and findings , 2005, Int. J. Approx. Reason..

[32]  Dalia Ghanem.  Testing Identifying Assumptions in Nonseparable Panel Data Models , 2017 .

[33]  Joseph P. Romano,et al.  Randomization Tests Under an Approximate Symmetry Assumption , 2017 .

[34]  J. Aislinn Bohren,et al.  Optimal Design of Experiments in the Presence of Interference , 2017, Review of Economics and Statistics.

[35]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[36]  Christian Hansen,et al.  Identification of Marginal Effects in a Nonparametric Correlated Random Effects Model , 2006 .

[37]  G. Imbens,et al.  Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score , 2000 .

[38]  J. Heckman.  The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models , 1976 .

[39]  Azeem M. Shaikh,et al.  Inference Under Covariate-Adaptive Randomization , 2015, Journal of the American Statistical Association.

[40]  M. Lechner,et al.  Endogeneity and Non-Response Bias in Treatment Evaluation: Nonparametric Identification of Causal Effects by Instruments , 2015, Journal of Applied Econometrics.

[41]  David J. McKenzie,et al.  Beyond Baseline and Follow-Up: The Case for More T in Experiments , 2011 .

[42]  David S. Lee.  Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects , 2005 .

[43]  Rosa L. Matzkin,et al.  Cross Section and Panel Data Estimators for Nonseparable Models with Endogenous Regressors , 2005 .

[44]  D. Rubin,et al.  Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction , 2016 .

[45]  Luc Behaghel,et al.  Estimating the Effect of Treatments Allocated by Randomized Waiting Lists , 2015, Econometrica.

[46]  Emmanuel Skoufias,et al.  PROGRESA and Its Impacts on the Welfare of Rural Households in Mexico , 2005 .

[47]  J. Robins,et al.  Estimation of Regression Coefficients When Some Regressors are not Always Observed , 1994 .

[48]  J. Horowitz,et al.  Nonparametric Analysis of Randomized Experiments with Missing Covariate and Outcome Data , 2000 .

[49]  Whitney K. Newey,et al.  Nonparametric Estimation of Sample Selection Models , 2003 .