Least Square Regularized Regression in Sum Space

This paper proposes a least square regularized regression algorithm in sum space of reproducing kernel Hilbert spaces (RKHSs) for nonflat function approximation, and obtains the solution of the algorithm by solving a system of linear equations. This algorithm can approximate the low- and high-frequency component of the target function with large and small scale kernels, respectively. The convergence and learning rate are analyzed. We measure the complexity of the sum space by its covering number and demonstrate that the covering number can be bounded by the product of the covering numbers of basic RKHSs. For sum space of RKHSs with Gaussian kernels, by choosing appropriate parameters, we tradeoff the sample error and regularization error, and obtain a polynomial learning rate, which is better than that in any single RKHS. The utility of this method is illustrated with two simulated data sets and five real-life databases.

[1]  Sergios Theodoridis,et al.  Adaptive Multiregression in Reproducing Kernel Hilbert Spaces: The Multiaccess MIMO Channel Case , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[2]  Yiming Ying,et al.  Multi-kernel regularized classifiers , 2007, J. Complex..

[3]  Yiming Ying,et al.  Learnability of Gaussians with Flexible Variances , 2007, J. Mach. Learn. Res..

[4]  S. Smale,et al.  ESTIMATING THE APPROXIMATION ERROR IN LEARNING THEORY , 2003 .

[5]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[6]  Francis R. Bach,et al.  Consistency of the group Lasso and multiple kernel learning , 2007, J. Mach. Learn. Res..

[7]  Ingo Steinwart,et al.  Fast Rates for Support Vector Machines , 2005, COLT.

[8]  Jiaxin Wang,et al.  Non-flat function estimation with a multi-scale support vector regression , 2006, Neurocomputing.

[9]  Haralambos Sarimveis,et al.  A new algorithm for online structure and parameter adaptation of RBF networks , 2003, Neural Networks.

[10]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[11]  Felipe Cucker,et al.  Best Choices for Regularization Parameters in Learning Theory: On the Bias—Variance Problem , 2002, Found. Comput. Math..

[12]  Hong Yan,et al.  Framelet Kernels With Applications to Support Vector Regression and Regularization Networks , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[14]  Sergios Theodoridis,et al.  Adaptive Learning in Complex Reproducing Kernel Hilbert Spaces Employing Wirtinger's Subgradients , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Di-Rong Chen,et al.  Partially-Linear Least-Squares Regularized Regression for System Identification , 2009, IEEE Transactions on Automatic Control.

[16]  Paramasivan Saratchandran,et al.  Performance evaluation of a sequential minimal radial basis function (RBF) neural network learning algorithm , 1998, IEEE Trans. Neural Networks.

[17]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[18]  Girish Chowdhary,et al.  A reproducing Kernel Hilbert Space approach for the online update of Radial Bases in neuro-adaptive control , 2011, IEEE Conference on Decision and Control and European Control Conference.

[19]  Zenglin Xu,et al.  Efficient Sparse Generalized Multiple Kernel Learning , 2011, IEEE Transactions on Neural Networks.

[20]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[21]  Tomaso A. Poggio,et al.  Regularization Networks and Support Vector Machines , 2000, Adv. Comput. Math..

[22]  Adrian G. Bors,et al.  Introduction of the Radial Basis Function (RBF) Networks , 2000 .

[23]  G. Wahba.  Spline models for observational data , 1990 .

[24]  Pedro Antonio Gutiérrez,et al.  Logistic Regression by Means of Evolutionary Radial Basis Function Neural Networks , 2011, IEEE Transactions on Neural Networks.

[25]  Yiming Ying,et al.  Learning Rates of Least-Square Regularized Regression , 2006, Found. Comput. Math..

[26]  Johan A. K. Suykens,et al.  Kernel based partially linear models and nonlinear identification , 2005, IEEE Transactions on Automatic Control.

[27]  Roland Opfer,et al.  Multiscale kernels , 2006, Adv. Comput. Math..

[28]  Jean-Philippe Vert,et al.  Consistency and Convergence Rates of One-Class SVMs and Related Algorithms , 2006, J. Mach. Learn. Res..

[29]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[30]  Felipe Cucker,et al.  Learning Theory: An Approximation Theory Viewpoint: On the bias–variance problem , 2007 .

[31]  Ke Meng,et al.  Self-adaptive radial basis function neural network for short-term electricity price forecasting , 2009 .

[32]  William Stafford Noble,et al.  Kernel methods for predicting protein-protein interactions , 2005, ISMB.

[33]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[34]  Dingli Yu,et al.  Adaptive RBF network for parameter estimation and stable air-fuel ratio control , 2008, Neural Networks.

[35]  Marco Sciandrone,et al.  Efficient training of RBF neural networks for pattern recognition , 2001, IEEE Trans. Neural Networks.

[36]  Alexander J. Smola,et al.  Learning the Kernel with Hyperkernels , 2005, J. Mach. Learn. Res..

[37]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[38]  Stéphane Canu,et al.  Frames, Reproducing Kernels, Regularization and Learning , 2005, J. Mach. Learn. Res..