Local regularization assisted orthogonal least squares regression

A locally regularized orthogonal least squares (LROLS) algorithm is proposed for constructing parsimonious or sparse regression models that generalize well. By associating each orthogonal weight in the regression model with an individual regularization parameter, the ability for the orthogonal least squares model selection to produce a very sparse model with good generalization performance is greatly enhanced. Furthermore, with the assistance of local regularization, when to terminate the model selection procedure becomes much clearer. A comparison with a state-of-the-art method for constructing sparse regression models, known as the relevance vector machine, is given. The proposed LROLS algorithm is shown to possess considerable computational advantages, including well conditioned solution and faster convergence speed.

[1]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[2]  Martin Brown,et al.  Neurofuzzy adaptive modelling and control , 1994 .

[3]  R. Harrison,et al.  Support Vector Machines for System Identification , 1998 .

[4]  Lajos Hanzo,et al.  Support vector machine multiuser receiver for DS-CDMA signals in multipath channels , 2001, IEEE Trans. Neural Networks.

[5]  Klaus Obermayer,et al.  Classi cation on Pairwise Proximity , 2007 .

[6]  D. Donoho,et al.  Basis pursuit , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[7]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[8]  R. C. Williamson,et al.  Classification on proximity data with LP-machines , 1999 .

[9]  Chris Bishop,et al.  Improving the Generalization Properties of Radial Basis Function Neural Networks , 1991, Neural Computation.

[10]  David J. C. MacKay,et al.  The Evidence Framework Applied to Classification Networks , 1992, Neural Computation.

[11]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[12]  Sheng Chen,et al.  Representations of non-linear systems: the NARMAX model , 1989 .

[13]  I. J. Leontaritis,et al.  Model selection and validation methods for non-linear systems , 1987 .

[14]  T. Kavli.  ASMO—Dan algorithm for adaptive spline modelling of observation data , 1993 .

[15]  Stephen A. Billings,et al.  Time series prediction using support vector machines, the orthogonal and the regularized orthogonal least-squares algorithms , 2002, Int. J. Syst. Sci..

[16]  Sheng Chen,et al.  Regularized orthogonal least squares algorithm for constructing radial basis function networks , 1996 .

[17]  Sheng Chen,et al.  Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks , 1999, IEEE Trans. Neural Networks.

[18]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[19]  S. Chen,et al.  Fast orthogonal least squares algorithm for efficient subset model selection , 1995, IEEE Trans. Signal Process..

[20]  Gene H. Golub,et al.  Matrix computations , 1983 .

[21]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[22]  George Eastman House,et al.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001 .

[23]  Sheng Chen,et al.  Extended model set, global data and threshold model identification of severely non-linear systems , 1989 .

[24]  S. A. Billings,et al.  The identification of linear and non-linear models of a turbocharged automotive diesel engine , 1989 .

[25]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[26]  O. Mangasarian,et al.  Robust linear programming discrimination of two linearly inseparable sets , 1992 .

[27]  Mark J. L. Orr.  Local Smoothing of Radial Basis Function Networks , 1995 .

[28]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.