Extended Kernel Recursive Least Squares Algorithm

This paper presents a kernelized version of the extended recursive least squares (EX-KRLS) algorithm which implements for the first time a general linear state model in reproducing kernel Hilbert spaces (RKHS), or equivalently a general nonlinear state model in the input space. The center piece of this development is a reformulation of the well known extended recursive least squares (EX-RLS) algorithm in RKHS which only requires inner product operations between input vectors, thus enabling the application of the kernel property (commonly known as the kernel trick). The first part of the paper presents a set of theorems that shows the generality of the approach. The EX-KRLS is preferable to 1) a standard kernel recursive least squares (KRLS) in applications that require tracking the state-vector of general linear state-space models in the kernel space, or 2) an EX-RLS when the application requires a nonlinear observation and state models. The second part of the paper compares the EX-KRLS in nonlinear Rayleigh multipath channel tracking and in Lorenz system modeling problem. We show that the proposed algorithm is able to outperform the standard KRLS and EX-RLS in both simulations.

[1]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[2]  Gavin C. Cawley,et al.  Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers , 2003, Pattern Recognit..

[3]  Ali H. Sayed,et al.  Adaptive Filters , 2008 .

[4]  Ingo Steinwart,et al.  On the Influence of the Kernel on the Consistency of Support Vector Machines , 2002, J. Mach. Learn. Res..

[5]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[8]  Lehel Csató,et al.  Sparse On-Line Gaussian Processes , 2002, Neural Computation.

[9]  Robert F. Harrison,et al.  A kernel based adaline , 1999, ESANN.

[10]  Shie Mannor,et al.  Sparse Online Greedy Support Vector Regression , 2002, ECML.

[11]  Ignacio Santamaría,et al.  A Sliding-Window Kernel RLS Algorithm and Its Application to Nonlinear Channel Identification , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[12]  Jeffrey K. Uhlmann,et al.  New extension of the Kalman filter to nonlinear systems , 1997, Defense, Security, and Sensing.

[13]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[14]  Jeffrey S. Racine,et al.  An efficient cross-validation algorithm for window width selection for nonparametric kernel regression , 1993 .

[15]  John W. Auer,et al.  Linear algebra with applications , 1996 .

[16]  S. Haykin,et al.  Kernel Least‐Mean‐Square Algorithm , 2010 .

[17]  Senjian An,et al.  Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression , 2007, Pattern Recognit..

[18]  Warwick Tucker,et al.  Foundations of Computational Mathematics a Rigorous Ode Solver and Smale's 14th Problem , 2022 .

[19]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[20]  L. Ralaivola,et al.  Time series filtering, smoothing and learning using the kernel Kalman filter , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[21]  Timothy J. Robinson,et al.  Sequential Monte Carlo Methods in Practice , 2003 .

[22]  Charles A. Micchelli,et al.  Learning the Kernel Function via Regularization , 2005, J. Mach. Learn. Res..

[23]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[24]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[25]  Charles A. Micchelli,et al.  Learning Convex Combinations of Continuously Parameterized Basic Kernels , 2005, COLT.

[26]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[27]  Liva Ralaivola,et al.  Dynamical Modeling with Kernels for Nonlinear Time Series Prediction , 2003, NIPS.

[28]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[29]  Rudolph van der Merwe,et al.  The unscented Kalman filter for nonlinear estimation , 2000, Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373).

[30]  M. Opper.  Sparse Online Gaussian Processes , 2008 .

[31]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[32]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[33]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[34]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[35]  Sayan Mukherjee,et al.  Choosing Multiple Parameters for Support Vector Machines , 2002, Machine Learning.

[36]  Anthony Kuh,et al.  Adaptive kernel methods for CDMA systems , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[37]  Paulo Sergio Ramirez,et al.  Fundamentals of Adaptive Filtering , 2002 .

[38]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[39]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[40]  R. Shah,et al.  Least Squares Support Vector Machines , 2022 .

[41]  Weifeng Liu,et al.  Kernel Affine Projection Algorithms , 2008, EURASIP J. Adv. Signal Process..