Return of the Phantom Menace

Scholars often assume that the danger posed by omitted variable bias can be ameliorated by the inclusion of large numbers of relevant control variables. However, there is nothing in the mathematics of regression analysis that supports this conclusion. This paper goes beyond textbook treatments of omitted variable bias and shows, both for OLS and for generalized linear models, that the inclusion of additional control variables may increase or decrease the bias, and we cannot know for sure which is the case in any particular situation. The last section of the paper shows how formal sensitivity analysis can be used to determine whether omitted variables are a problem. A substantive example demonstrates the method.

[1]  R. Buckley.  Speak softly and carry a big stick , 2000 .

[2]  Q. Vuong.  Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses , 1989 .

[3]  Tanja Ellingsen,et al.  Toward a Democratic Civil Peace? Democracy, Political Change, and Civil War, 1816–1992 , 2001, American Political Science Review.

[4]  Kevin A. Clarke.  A Simple Distribution-Free Test for Nonnested Hypotheses † , 2003 .

[5]  Jan Kmenta,et al.  Elements of econometrics , 1988 .

[6]  M. Gail,et al.  Biased estimates of treatment effect in randomized experiments with nonlinear regressions and omitted covariates , 1984 .

[7]  C. Sims.  MACROECONOMICS AND REALITY , 1977 .

[8]  Peter Schmidt,et al.  The Theory and Practice of Econometrics , 1985 .

[9]  Kevin A. Clarke.  Testing Nonnested Models of International Relations: Reevaluating Realism , 2001 .

[10]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[11]  G. Judge,et al.  The Theory and Practice of Econometrics , 1981 .

[12]  E. C. Hammond,et al.  Smoking and lung cancer: recent evidence and a discussion of some questions. , 1959, Journal of the National Cancer Institute.

[13]  G. W. Imbens.  Sensitivity to Exogeneity Assumptions in Program Evaluation , 2003 .

[14]  Henry E. Brady,et al.  THE COSTS OF VOTING: EVIDENCE FROM A NATURAL EXPERIMENT , 2004 .

[15]  Christopher H. Achen.  Let's Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong , 2005 .

[16]  D. Rubin,et al.  Assessing Sensitivity to an Unobserved Binary Covariate in an Observational Study with Binary Outcome , 1983 .

[17]  J. MacKinnon,et al.  Estimation and inference in econometrics , 1994 .

[18]  E. Hanushek.  Statistical methods for social scientists , 1977 .

[19]  Jeffrey M. Woodbridge.  Econometric Analysis of Cross Section and Panel Data , 2002 .

[20]  Colin L. Mallows,et al.  Some Comments on Cp , 2000, Technometrics.

[21]  J. Neyman,et al.  INADMISSIBILITY OF THE USUAL ESTIMATOR FOR THE MEAN OF A MULTIVARIATE NORMAL DISTRIBUTION , 2005 .

[22]  Robert O. Keohane,et al.  Designing Social Inquiry: Scientific Inference in Qualitative Research. , 1995 .

[23]  T. Rudolph,et al.  Political Trust, Ideology, and Public Support for Government Spending , 2005 .

[24]  S. Gates,et al.  The Geography of Civil War , 2002 .

[25]  Kevin A. Clarke.  Nonparametric Model Discrimination in International Relations , 2003 .

[26]  Lillian Cohen,et al.  Statistical Methods for Social Scientists. , 1954 .

[27]  A. Gerber,et al.  Equal Votes, Equal Money: Court-Ordered Redistricting and Public Expenditures in the American States , 2002, American Political Science Review.

[28]  Kevin A. Clarke.  A Simple Distribution-Free Test for Nonnested Model Selection , 2007, Political Analysis.

[29]  George A. Krause.  Coping with Uncertainty: Analyzing Risk Propensities of SEC Budgetary Decisions, 1949–97 , 2003, American Political Science Review.

[30]  P. McCullagh,et al.  Generalized Linear Models, 2nd Edn. , 1990 .

[31]  Michael A. Bailey,et al.  Signals from the Tenth Justice: The Political Role of the Solicitor General in Supreme Court Decision Making , 2005 .

[32]  Bruce D. Meyer.  Natural and Quasi- Experiments in Economics , 1994 .

[33]  Christopher H. Achen.  Social psychology, demographic variables, and linear regression: Breaking the iron triangle in voting research , 1992 .

[34]  Z. Griliches.  Estimating the Returns to Schooling: Some Econometric Problems. , 1977 .

[35]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[36]  John DiNardo,et al.  Econometric methods. 4th ed. , 1997 .

[37]  Russell D. Murphy,et al.  TOWARD A NEW POLITICAL METHODOLOGY: Microfoundations and ART , 2006 .

[38]  T. Shakespeare,et al.  Observational Studies , 2003 .

[39]  Kevin A. Clarke.  The Phantom Menace: Omitted Variable Bias in Econometric Research , 2005 .

[40]  G. King,et al.  Analyzing Incomplete Political Science Data: An Alternative Algorithm for Multiple Imputation , 2001, American Political Science Review.

[41]  P. Rosenbaum.  Dropping out of High School in the United States: An Observational Study , 1986 .

[42]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[43]  Christopher R. Taber,et al.  Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools , 2000, Journal of Political Economy.

[44]  P. Holland.  Statistics and Causal Inference , 1985 .

[45]  L. Breiman.  The Little Bootstrap and other Methods for Dimensionality Selection in Regression: X-Fixed Prediction Error , 1992 .

[46]  Finis Welch.  Human Capital Theory: Education, Discrimination, and Lifecycles, , 1975 .

[47]  N. Jewell,et al.  A geometric approach to assess bias due to omitted covariates in generalized linear models , 1993 .

[48]  David F. Hendry,et al.  On the Formulation of Empirical-models in Dynamic Econometrics , 1982 .

[49]  P. Feaver,et al.  Speak Softly and Carry a Big Stick? Veterans in the Political Elite and the American Use of Force , 2002, American Political Science Review.

[50]  D. Cox.  Tests of Separate Families of Hypotheses , 1961 .

[51]  A. Buse,et al.  Elements of econometrics , 1972 .