Kernel Normalised Least Mean Squares with Delayed Model Adaptation

Kernel adaptive filters (KAFs) are non-linear filters which can adapt temporally and have the additional benefit of being computationally efficient through use of the “kernel trick”. In a number of real-world applications, such as channel equalisation, the non-linear mapping provides significant improvements over conventional linear techniques such as the least mean squares (LMS) and recursive least squares (RLS) algorithms. Prior works have focused mainly on the theory and accuracy of KAFs, with little research on their implementations. This article proposes several variants of algorithms based on the kernel normalised least mean squares (KNLMS) algorithm which utilise a delayed model update to minimise dependencies. Subsequently, this work proposes corresponding hardware architectures which utilise this delayed model update to achieve high sample rates and low latency while also providing high modelling accuracy. The resultant delayed KNLMS (DKNLMS) algorithms can achieve clock rates up to 12× higher than the standard KNLMS algorithm, with minimal impact on accuracy and stability. A system implementation achieves 250 GOps/s and a throughput of 187.4 MHz on an Ultra96 board with 1.8× higher throughput than previous state of the art.

[1]  Alexander J. Smola,et al.  Fastfood: Approximate Kernel Expansions in Loglinear Time , 2014, ArXiv.

[2]  R.D. Poltmann,et al.  Conversion of the delayed LMS algorithm into the LMS algorithm , 1995, IEEE Signal Processing Letters.

[3]  Nanning Zheng,et al.  Survival kernel with application to kernel adaptive filtering , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[4]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[5]  Barbara Hammer,et al.  A Note on the Universal Approximation Capability of Support Vector Machines , 2003, Neural Processing Letters.

[6]  Feng Ding,et al.  Decomposition based fast least squares algorithm for output error systems , 2013, Signal Process..

[7]  Craig T. Jin,et al.  A fully pipelined kernel normalised least mean squares processor for accelerated parameter optimisation , 2015, 2015 25th International Conference on Field Programmable Logic and Applications (FPL).

[8]  Craig T. Jin,et al.  Random projections for scaling machine learning on FPGAs , 2016, 2016 International Conference on Field-Programmable Technology (FPT).

[9]  Philip Heng Wai Leong,et al.  A Microcoded Kernel Recursive Least Squares Processor Using FPGA Technology , 2016, ACM Trans. Reconfigurable Technol. Syst..

[10]  Scott C. Douglas,et al.  A pipelined LMS adaptive FIR filter architecture without adaptation delay , 1998, IEEE Trans. Signal Process..

[11]  Avi Mendelson,et al.  UNIQ , 2018, ACM Trans. Comput. Syst..

[12]  Ignacio Santamaría,et al.  A Sliding-Window Kernel RLS Algorithm and Its Application to Nonlinear Channel Identification , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[13]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[14]  Qian Wang,et al.  AUGEM: Automatically generate high performance Dense Linear Algebra kernels on x86 CPUs , 2013, 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[15]  Paul Honeine,et al.  Online Prediction of Time Series Data With Kernels , 2009, IEEE Transactions on Signal Processing.

[16]  Gerald Estrin,et al.  Organization of computer systems: the fixed plus variable structure computer , 1960, IRE-AIEE-ACM '60 (Western).

[17]  Fuyun Ling,et al.  The LMS algorithm with delayed coefficient adaptation , 1989, IEEE Trans. Acoust. Speech Signal Process..

[18]  Alexander J. Smola,et al.  Online learning with kernels , 2001, IEEE Transactions on Signal Processing.

[19]  R. C. Whaley,et al.  Minimizing development and maintenance costs in supporting persistently optimized BLAS , 2005, Softw. Pract. Exp..

[20]  Davide Anguita,et al.  A Hardware-friendly Support Vector Machine for Embedded Automotive Applications , 2007, 2007 International Joint Conference on Neural Networks.

[21]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[22]  Shie Mannor,et al.  The kernel recursive least-squares algorithm , 2004, IEEE Transactions on Signal Processing.

[23]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[24]  Fuad E. Alsaadi,et al.  Iterative parameter identification for pseudo-linear systems with ARMA noise using the filtering technique , 2018 .

[25]  Christopher M. Bishop,et al.  Current address: Microsoft Research, , 2022 .

[26]  Craig T. Jin,et al.  FPGA Implementations of Kernel Normalised Least Mean Squares Processors , 2017, TRETS.

[27]  Weifeng Liu,et al.  Fixed-budget kernel recursive least-squares , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[28]  Philip Heng Wai Leong,et al.  Braiding: A scheme for resolving hazards in kernel adaptive filters , 2015, 2015 International Conference on Field Programmable Technology (FPT).

[29]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[30]  Weifeng Liu,et al.  Kernel Adaptive Filtering: A Comprehensive Introduction , 2010 .

[31]  Philip Heng Wai Leong,et al.  FPGA Fastfood - A High Speed Systolic Implementation of a Large Scale Online Kernel Method , 2018, FPGA.

[32]  A. Hall,et al.  Adaptive Switching Circuits , 2016 .

[33]  Lok-Kee Ting,et al.  High Speed FPGA-Based Implementations of Delayed-LMS Filters , 2005, J. VLSI Signal Process..

[34]  John Wawrzynek,et al.  Chisel: Constructing hardware in a Scala embedded language , 2012, DAC Design Automation Conference 2012.

[35]  Avi Mendelson,et al.  NICE: Noise Injection and Clamping Estimation for Neural Network Quantization , 2018, Mathematics.

[36]  BengioYoshua,et al.  Random search for hyper-parameter optimization , 2012 .

[37]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[38]  Nanning Zheng,et al.  Hardware implementation of KLMS algorithm using FPGA , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).