Statistical Inference After Model Selection

Conventional statistical inference requires that a model of how the data were generated be known before the data are analyzed. Yet in criminology, and in the social sciences more broadly, a variety of model selection procedures are routinely undertaken followed by statistical tests and confidence intervals computed for a “final” model. In this paper, we examine such practices and show how they are typically misguided. The parameters being estimated are no longer well defined, and post-model-selection sampling distributions are mixtures with properties that are very different from what is conventionally assumed. Confidence intervals and statistical tests do not perform as they should. We examine in some detail the specific mechanisms responsible. We also offer some suggestions for better practice and show though a criminal justice example using real data how proper statistical inference in principle may be obtained.

[1]  Chester Hartman,et al.  Rejoinder by the Author , 1965 .

[2]  L. Brown.  THE CONDITIONAL LEVEL OF STUDENT'S t TEST' , 1967 .

[3]  R. Olshen.  The Conditional Level of the F—Test , 1973 .

[4]  G. Box.  Science and Statistics , 1976 .

[5]  Edward E. Leamer,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[6]  J. Richard,et al.  Specification Searches: Ad Hoc Inference with Nonexperimental Data , 1980 .

[7]  V. Barnett,et al.  Comparative Statistical Inference (2nd ed.). , 1983 .

[8]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[9]  D. J. Harris,et al.  Research on Sentencing: The Search for Reform , 1984 .

[10]  D. Rubin.  Statistics and Causal Inference: Comment: Which Ifs Have Causal Answers , 1986 .

[11]  D. Rubin.  Comment: Which Ifs Have Causal Answers , 1986 .

[12]  D. Freedman.  As Others See Us: A Case Study in Path Analysis , 1987 .

[13]  Theo K. Dijkstra,et al.  On Model Uncertainty and its Statistical Implications , 1988 .

[14]  D. Freedman,et al.  On the Impact of Variable Selection in Fitting Regression Equations , 1988 .

[15]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[16]  C. Manski.  Nonparametric Bounds on Treatment Effects , 1989 .

[17]  P. McCullagh,et al.  Generalized Linear Models, 2nd Edn. , 1990 .

[18]  N. Morris,et al.  Between Prison and Probation: Intermediate Punishments in a Rational Sentencing System , 1990 .

[19]  L. Brown.  An Ancillarity Paradox Which Appears in Multiple Linear Regression , 1990 .

[20]  J. Petersilia,et al.  Probation in the United States , 1997, Crime and Justice.

[21]  Vic Barnett,et al.  Comparative Statistical Inference , 1974, Technometrics.

[22]  J. Brian Gray,et al.  Applied Regression Including Computing and Graphics , 1999, Technometrics.

[23]  Y. Benjamini,et al.  THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY , 2001 .

[24]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001 .

[25]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[26]  H. Leeb,et al.  CAN ONE ESTIMATE THE UNCONDITIONAL DISTRIBUTION OF POST-MODEL-SELECTION ESTIMATORS? , 2003, Econometric Theory.

[27]  R. Berk.  Regression Analysis: A Constructive Critique , 2003 .

[28]  S. Raudenbush,et al.  Seeing Disorder: Neighborhood Stigma and the Social Construction of “Broken Windows” , 2004 .

[29]  D. Freedman.  Graphical models for causation, and the identification problem. , 2004, Evaluation review.

[30]  B. Efron.  Correlation and Large-Scale Simultaneous Significance Testing , 2007 .

[31]  B. M. Pötscher,et al.  MODEL SELECTION AND INFERENCE: FACTS AND FICTION , 2005, Econometric Theory.

[32]  J. Wooldredge,et al.  Un)anticipated Effects of Sentencing Reform on the Disparate Treatment of Defendants , 2005 .

[33]  David A. Freedman,et al.  Statistical Models: Theory and Practice: References , 2005 .

[34]  R. Lalonde,et al.  The Impact of Incarceration in State Prison on the Employment Prospects of Women , 2005, SSRN Electronic Journal.

[35]  J. Laub.  EDWIN H. SUTHERLAND AND THE MICHAEL‐ADLER REPORT: SEARCHING FOR THE SOUL OF CRIMINOLOGY SEVENTY YEARS LATER* , 2006 .

[36]  Garth Davies,et al.  VIOLENCE RISK SCREENING IN COMMUNITY CORRECTIONS , 2006 .

[37]  B. Johnson.  THE MULTILEVEL CONTEXT OF CRIMINAL SENTENCING: INTEGRATING JUDGE- AND COUNTY-LEVEL INFLUENCES * , 2006 .

[38]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[39]  Stephen A. Cernkovich,et al.  DRUG USE AND DESISTANCE PROCESSES , 2007 .

[40]  R. Tibshirani,et al.  Discussion: The Dantzig selector: Statistical estimation when p is much larger than n , 2007, 0803.3126.

[41]  Christopher Winship,et al.  Counterfactuals and Causal Inference: Methods and Principles for Social Research , 2007 .

[42]  Graham C. Ousey,et al.  Déjà vu All Over Again: Investigating Temporal Continuity of Adolescent Victimization , 2008 .

[43]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.