What to Do When Your Hessian is Not Invertible

What should a researcher do when statistical analysis software terminates before completion with a message that the Hessian is not invertible? The standard textbook advice is to respecify the model, but this is another way of saying that the researcher should change the question being asked. Obviously, however, computer programs should not be in the business of deciding what questions are worthy of study. Although noninvertable Hessians are sometimes signals of poorly posed questions, nonsensical models, or inappropriate estimators, they also frequently occur when information about the quantities of interest exists in the data through the likelihood function. The authors explain the problem in some detail and lay out two preliminary proposals for ways of dealing with noninvertable Hessians without changing the question asked.

[1]  J. Riley.  Solving systems of linear equations with a positive definite, symmetric, but possibly ill-conditioned matrix , 1955 .

[2]  G. King,et al.  Unifying Political Methodology: The Likelihood Theory of Statistical Inference , 1989 .

[3]  Philip E. Gill,et al.  Practical optimization , 1981 .

[4]  Leland Gerson Neuberg,et al.  A solution to the ecological inference problem: Reconstructing individual behavior from aggregate data , 1999 .

[5]  J. MacKinnon,et al.  Estimation and inference in econometrics , 1994 .

[6]  J. Berger.  Admissible Minimax Estimation of a Multivariate Normal Mean with Arbitrary Quadratic Loss , 1976 .

[7]  J. Fox.  Bootstrapping Regression Models , 2002 .

[8]  G. C. Wei,et al.  A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms , 1990 .

[9]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[10]  H. A. Schulke.  Matrix factorization , 1955, IRE Transactions on Circuit Theory.

[11]  M. J. D. Powell,et al.  Nonlinear Programming—Sequential Unconstrained Minimization Techniques , 1969 .

[12]  Douglas Rivers.  A Solution to the Ecological Inference Problem: Reconstructing Individual Behavior from Aggregate Data . By Gary King. Princeton, NJ: Princeton University Press, 1997. 342p. $55.00 cloth, $16.95 paper. , 1998 .

[13]  Alan J. Mayne,et al.  Generalized Inverse of Matrices and its Applications , 1972 .

[14]  Philip E. Gill,et al.  Newton-type methods for unconstrained and linearly constrained optimization , 1974, Math. Program..

[15]  Jason Wittenberg,et al.  Making the Most Of Statistical Analyses: Improving Interpretation and Presentation , 2000 .

[16]  Elizabeth Eskow,et al.  A New Modified Cholesky Factorization , 1990, SIAM J. Sci. Comput..

[17]  Edward Leamer.  Multicollinearity: A Bayesian Interpretation , 1973 .

[18]  Gary Smith,et al.  A Critique of Some Ridge Regression Methods , 1980 .

[19]  Donald W. Davies,et al.  A Comparison of Modified Newton Methods for Unconstrained Optimisation , 1971, Comput. J..

[20]  G. King,et al.  Listwise Deletion is Evil: What to Do About Missing Data in Political Science , 1998 .

[21]  William E. Strawderman,et al.  Minimax Adaptive Generalized Ridge Regression Estimators , 1978 .

[22]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .

[23]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[24]  Eric R. Ziegel,et al.  Generalized Linear Models , 2002, Technometrics.

[25]  Donald W. Marquaridt.  Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation , 1970 .

[26]  K. S. Banerjee.  Generalized Inverse of Matrices and Its Applications , 1973 .

[27]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[28]  B. Efron,et al.  Limiting the Risk of Bayes and Empirical Bayes Estimators—Part II: The Empirical Bayes Case , 1972 .

[29]  D. Sengupta.  Linear models , 2003 .

[30]  P. A. V. B. Swamy,et al.  Estimation of Linear Models with Time and Cross-Sectionally Varying Coefficients , 1977 .

[31]  J. Shao,et al.  The jackknife and bootstrap , 1996 .