LINEAR REGRESSION , WITH A COMMENTARY ON RIDGE REGRESSION Is MIXED ESTIMATION AND BAYESIAN METHODS

In this article, we discuss ways of using "dummy data" and mixed estimation (Theil and Goldberger, 1961) to bring external information formally into linear regression problems when the experimental data/model are inadequate. This is a useful way of attacking the same practical problems that motivated the development of ridge regression (Hoerl and Kennard, 1970). The main practical issues considered are (i) what form should the "dummy data" take?, and (ii) how much weight should it be given relative to the experimental data? When specific prior information is unavailable, it is suggested that the dummy data should reflect a preference for "stable" response functions and it is shown how this can be accomplished. Guidelines for the choice of the weighting parameter k (equivalent to the choice of the ridge parameter in ridge regression) are given. Upper limits for k are based on various tests of compatibility between the external (dummy) data and the experimental data. Lower limits for k are determined by the inadequacy of the data/model for the purpose(s) of the analysis. Finally, the parallel Bayesian approach is discussed, with emphasis on Box's (1980) framework of model estimation and criticism. AMS(MOS) Subject Classification: 62J99

[1]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[2]  P. Laycock,et al.  Some Criticisms of Stochastic Shrinkage and Ridge Regression, With Counterexamples , 1981 .

[3]  George E. P. Box,et al.  Sampling and Bayes' inference in scientific modelling and robustness , 1980 .

[4]  Gary Smith,et al.  A Critique of Some Ridge Regression Methods , 1980 .

[5]  Gary Smith.  An example of ridge regression difficulties , 1980 .

[6]  Norman R. Draper,et al.  Ridge Regression and James-Stein Estimation: Review and Comments , 1979 .

[7]  Robert L. Obenchain,et al.  Classical F-Tests and Confidence Regions for Ridge Regression , 1977 .

[8]  N. Wermuth,et al.  A Simulation Study of Alternatives to Ordinary Least Squares , 1977 .

[9]  R. R. Hocking.  The analysis and selection of variables in linear regression , 1976 .

[10]  J. F. Gentleman,et al.  Detecting Outliers. II. Supplementing the Direct Analysis of Residuals , 1975 .

[11]  M. Wilk,et al.  Detecting Outliers in a Two-Way Table: I. Statistical Behavior of Residuals* , 1975 .

[12]  A. E. Hoerl,et al.  Ridge regression:some simulations , 1975 .

[13]  L. Kupper,et al.  Experimental Design Considerations Based on a New Approach to Mean Square Error Estimation of Response Surfaces , 1974 .

[14]  G. C. Tiao,et al.  Bayesian inference in statistical analysis , 1973 .

[15]  D. Lindley,et al.  Bayes Estimates for the Linear Model , 1972 .

[16]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[17]  A. R. Manson,et al.  Minimum Bias Estimation and Experimental Design for Response Surfaces , 1969 .

[18]  H. Theil.  On the Use of Incomplete Prior Information in Regression Analysis , 1963 .

[19]  A. Goldberger,et al.  On Pure and Mixed Statistical Estimation in Economics , 1961 .