Adaptive Learning in Cartesian Product of Reproducing Kernel Hilbert Spaces

We propose a novel adaptive learning algorithm based on iterative orthogonal projections in the Cartesian product of multiple reproducing kernel Hilbert spaces (RKHSs). The objective is to estimate or track nonlinear functions that are supposed to contain multiple components such as i) linear and nonlinear components and ii) high- and low- frequency components. In this case, the use of multiple RKHSs permits a compact representation of multicomponent functions. The proposed algorithm is where two different methods of the author meet: multikernel adaptive filtering and the algorithm of hyperplane projection along affine subspace (HYPASS). In a particular case, the “sum” space of the RKHSs is isomorphic, under a straightforward correspondence, to the product space, and hence the proposed algorithm can also be regarded as an iterative projection method in the sum space. The efficacy of the proposed algorithm is shown by numerical examples.

[1]  Masahiro Yukawa.  Cartesian Multikernel Adaptive Filtering (信号処理) , 2014 .

[2]  Akira Tanaka,et al.  Theoretical analyses on a class of nested RKHS's , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Yuichi Motai,et al.  Kernel Association for Classification and Prediction: A Survey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Paul Honeine,et al.  Online Prediction of Time Series Data With Kernels , 2009, IEEE Transactions on Signal Processing.

[5]  Marc Moonen,et al.  Nonlinear Acoustic Echo Cancellation Based on a Sliding-Window Leaky Kernel Affine Projection Algorithm , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[6]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[7]  Masahiro Yukawa,et al.  Multikernel Adaptive Filtering , 2012, IEEE Transactions on Signal Processing.

[8]  J. Nagumo,et al.  A learning method for system identification , 1967, IEEE Transactions on Automatic Control.

[9]  Johan A. K. Suykens,et al.  Kernel based partially linear models and nonlinear identification , 2005, IEEE Transactions on Automatic Control.

[10]  Lu Liu,et al.  Least Square Regularized Regression in Sum Space , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[11]  Sergios Theodoridis,et al.  Adaptive Learning in a World of Projections , 2011, IEEE Signal Processing Magazine.

[12]  Alexander J. Smola,et al.  Online learning with kernels , 2001, IEEE Transactions on Signal Processing.

[13]  Masahiro Yukawa,et al.  An efficient kernel adaptive filtering algorithm using hyperplane projection along affine subspace , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[14]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[15]  W. T. Federer,et al.  Stochastic Approximation and NonLinear Regression , 2003 .

[16]  Jean-Philippe Vert,et al.  Consistency and Convergence Rates of One-Class SVMs and Related Algorithms , 2006, J. Mach. Learn. Res..

[17]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[18]  D. Meeter.  Stochastic Approximation and Nonlinear Regression , 1969 .

[19]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[20]  Jie Chen,et al.  Online Dictionary Learning for Kernel LMS , 2014, IEEE Transactions on Signal Processing.

[21]  Masahiro Yukawa,et al.  An efficient sparse kernel adaptive filtering algorithm based on isomorphism between functional subspace and Euclidean space , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Wolfgang Härdle,et al.  Partially Linear Models , 2000 .

[23]  Badong Chen,et al.  Quantized Kernel Least Mean Square Algorithm , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Cédric Richard,et al.  Convex combinations of kernel adaptive filters , 2014, 2014 IEEE International Workshop on Machine Learning for Signal Processing (MLSP).

[25]  Yih-Fang Huang,et al.  Kernelized set-membership approach to nonlinear adaptive filtering , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[26]  Sun-Yuan Kung,et al.  Multikernel Least Mean Square Algorithm , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[27]  I. Yamada,et al.  Adaptive Projected Subgradient Method for Asymptotic Minimization of Sequence of Nonnegative Convex Functions , 2005 .

[28]  Weifeng Liu,et al.  Kernel Adaptive Filtering , 2010 .

[29]  Isao Yamada,et al.  A Unified View of Adaptive Variable-Metric Projection Algorithms , 2009, EURASIP J. Adv. Signal Process..

[30]  José Carlos Príncipe,et al.  Mixture kernel least mean square , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[31]  Sergios Theodoridis,et al.  Adaptive Constrained Learning in Reproducing Kernel Hilbert Spaces: The Robust Beamforming Case , 2009, IEEE Transactions on Signal Processing.

[32]  Miguel Lázaro-Gredilla,et al.  Kernel Recursive Least-Squares Tracker for Time-Varying Regression , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Toshihisa Tanaka,et al.  Multikernel adaptive filters with multiple dictionaries and regularization , 2013, 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.

[34]  Don R. Hush,et al.  An Explicit Description of the Reproducing Kernel Hilbert Spaces of Gaussian RBF Kernels , 2006, IEEE Transactions on Information Theory.

[35]  H. Minh,et al.  Some Properties of Gaussian Reproducing Kernel Hilbert Spaces and Their Implications for Function Approximation and Learning Theory , 2010 .

[36]  Masahiro Yukawa.  Nonlinear adaptive filtering techniques with multiple kernels , 2011, 2011 19th European Signal Processing Conference.

[37]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[38]  Weifeng Liu,et al.  Kernel Affine Projection Algorithms , 2008, EURASIP J. Adv. Signal Process..

[39]  Masahiro Yukawa,et al.  On adaptivity of online model selection method based on multikernel adaptive filtering , 2013, 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.

[40]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[41]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[42]  齋藤 三郎.  Integral transforms, reproducing kernels and their applications , 1997 .

[43]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[44]  Masahiro Yukawa.  Online learning based on iterative projections in sum space of linear and Gaussian reproducing kernel Hilbert spaces , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[45]  Masahiro Yukawa,et al.  Online model selection and learning by multikernel adaptive filtering , 2013, 21st European Signal Processing Conference (EUSIPCO 2013).

[46]  Sergios Theodoridis,et al.  Special Issue on Advances in Kernel-Based Learning for Signal Processing , 2013 .

[47]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[48]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[49]  Masahiro Yukawa,et al.  Adaptive Nonlinear Estimation Based on Parallel Projection Along Affine Subspaces in Reproducing Kernel Hilbert Space , 2015, IEEE Transactions on Signal Processing.

[50]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[51]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[52]  Di-Rong Chen,et al.  Partially-Linear Least-Squares Regularized Regression for System Identification , 2009, IEEE Transactions on Automatic Control.

[53]  Sergios Theodoridis,et al.  Online Learning in Reproducing Kernel Hilbert Spaces , 2014 .

[54]  Badong Chen,et al.  Online efficient learning with quantized KLMS and L1 regularization , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[55]  Masahiro Yukawa,et al.  An efficient data-reusing kernel adaptive filtering algorithm based on Parallel HYperslab Projection along Affine Subspaces , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[56]  Sergios Theodoridis,et al.  Online Kernel-Based Classification Using Adaptive Projection Algorithms , 2008, IEEE Transactions on Signal Processing.