Ridge regression: biased estimation for nonorthogonal problems

In multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory, if not incorrect, if the prediction vectors are not orthogonal. Proposed is an estimation procedure based on adding small positive quantities to the diagonal of X′X. Introduced is the ridge trace, a method for showing in two dimensions the effects of nonorthogonality. It is then shown how to augment X′X to obtain biased estimates with smaller mean square error.

[1]  H. Jeffreys,et al.  Theory of probability , 1896 .

[2]  J. Riley.  Solving systems of linear equations with a positive definite, symmetric, but possibly ill-conditioned matrix , 1955 .

[3]  David R. Cox,et al.  The Analysis of Variance , 1960 .

[4]  H. Scheffé,et al.  The Analysis of Variance , 1960 .

[5]  H. Raiffa,et al.  Applied Statistical Decision Theory. , 1961 .

[6]  C. Stein.  Confidence Sets for the Mean of a Multivariate Normal Distribution , 1962 .

[7]  Arthur E. Hoerl,et al.  Application of ridge analysis to regression problems , 1962 .

[8]  A. Balakrishnant.  An Operator Theoretic Formulation of a Class of Control Problems and a Steepest Descent Method of Solution , 1963 .

[9]  G. Barnard.  The Logic of Least Squares , 1963 .

[10]  M. Clutton-Brock.  Using the Observations to Estimate the Prior Distribution , 1965 .

[11]  M. J. Garside,et al.  The Best Sub‐Set in Multiple Regression Analysis , 1965 .

[12]  John T. Scott.  Factor Analysis and Regression , 1966 .

[13]  J. W. Gorman,et al.  Selection of Variables for Fitting Equations to Data , 1966 .

[14]  R. R. Hocking,et al.  Selection of the Best Subset in Regression Analysis , 1967 .

[15]  M. Kendall,et al.  The discarding of variables in multivariate analysis. , 1967, Biometrika.

[16]  J. N. R. Jeffers,et al.  Two Case Studies in the Application of Principal Component Analysis , 1967 .

[17]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[18]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .