Comparison of the Convergence Rates of the New Correntropy-Based Levenberg–Marquardt (CLM) Method and the Fixed-Point Maximum Correntropy (FP-MCC) Algorithm

Correntropy as an efficient information theoretic (ITL) criterion has been extensively applied in many non-Gaussian applications. In order to maximize correntropy, several optimization algorithms have been proposed. Fixed-point maximum correntropy (FP-MCC) and correntropy-based Levenberg–Marquardt (CLM) are the fastest of the proposed methods. As the convergence rate of these operational methods has not been studied before, in this paper we prove the ability of their quadratic convergence from a theoretic point of view and establish the influence of bandwidth on their convergence order. Then, theoretic results are validated through numerical experiments.

[1]  Deniz Erdogmus,et al.  A mutual information extension to the matched filter , 2005, Signal Process..

[2]  G. Franklin,et al.  An adaptive Gauss-Newton algorithm for training multilayer nonlinear filters that have embedded memory , 1999 .

[3]  Rate of convergence of the generalized newton algorithm using the fixed-point approach , 1984 .

[4]  John W. Fisher,et al.  Learning from Examples with Information Theoretic Criteria , 2000, J. VLSI Signal Process..

[5]  Weifeng Liu,et al.  Correntropy: A Localized Similarity Measure , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[6]  Dan Simon,et al.  Kalman filtering based on the maximum correntropy criterion in the presence of non-Gaussian noise , 2016 .

[7]  Tai-Kuo Woo,et al.  A Novel Convergence Accelerator for the LMS Adaptive Filter , 2012, Circuits Syst. Signal Process..

[8]  Ya-Xiang Yuan,et al.  On the Quadratic Convergence of the Levenberg-Marquardt Method without Nonsingularity Assumption , 2005, Computing.

[9]  Fuwen Yang,et al.  $H_{\infty }$ Fault Detection for Networked Mechanical Spring-Mass Systems With Incomplete Information , 2016, IEEE Transactions on Industrial Electronics.

[10]  Mojtaba Hajiabadi,et al.  Adaptive multitask network based on maximum correntropy learning algorithm , 2017 .

[11]  Jiandong Duan,et al.  Sparse Least Logarithmic Absolute Difference Algorithm with Correntropy-Induced Metric Penalty , 2016, Circuits Syst. Signal Process..

[12]  Ran He,et al.  Robust Principal Component Analysis Based on Maximum Correntropy Criterion , 2011, IEEE Transactions on Image Processing.

[13]  J. Heinonen.  Lectures on Lipschitz analysis , 2005 .

[14]  Majid Mohammadi,et al.  A robust Correntropy-based method for analyzing multisample aCGH data. , 2015, Genomics.

[15]  Jose C. Principe,et al.  Information Theoretic Learning - Renyi's Entropy and Kernel Perspectives , 2010, Information Theoretic Learning.

[16]  Saeed Khorashadizadeh,et al.  Adaptive fourier series-based control of electrically driven robot manipulators , 2013, The 3rd International Conference on Control, Instrumentation, and Automation.

[17]  Ran He,et al.  Two-Stage Nonnegative Sparse Representation for Large-Scale Face Recognition , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Nanning Zheng,et al.  Convergence of a Fixed-Point Algorithm under Maximum Correntropy Criterion , 2015, IEEE Signal Processing Letters.

[19]  E. Kamen.  A recursive parameter estimator yielding exponential convergence under sufficient excitation , 1989 .

[20]  Zongze Wu,et al.  Robust Hammerstein Adaptive Filtering under Maximum Correntropy Criterion , 2015, Entropy.

[21]  S. Banach.  Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales , 1922 .

[22]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[23]  P. P. Pokharel,et al.  Error Entropy, Correntropy and M-Estimation , 2006, 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing.

[24]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[25]  Jinyan Fan,et al.  The modified Levenberg-Marquardt method for nonlinear equations with cubic convergence , 2012, Math. Comput..

[26]  Weifeng Liu,et al.  Correntropy: Properties and Applications in Non-Gaussian Signal Processing , 2007, IEEE Transactions on Signal Processing.

[27]  Marc André Armand,et al.  On Single-Carrier Communication in Additive White Symmetric Alpha-Stable Noise , 2014, IEEE Transactions on Communications.

[28]  Denis G. Fantinato,et al.  Analysis of ITL Criteria in the Context of FIR Channel Equalization , 2016 .

[29]  Sheng Zhang,et al.  Adaptive Filtering Under a Variable Kernel Width Maximum Correntropy Criterion , 2017, IEEE Transactions on Circuits and Systems II: Express Briefs.

[30]  Badong Chen,et al.  Two Improved Normalized Subband Adaptive Filter Algorithms with Good Robustness Against Impulsive Interferences , 2015, Circuits, Systems, and Signal Processing.

[31]  Saeed Khorashadizadeh,et al.  Robust control of electrically driven robots by adaptive fuzzy estimation of uncertainty , 2012 .

[32]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[33]  Lu Lu,et al.  Active impulsive noise control using maximum correntropy with adaptive kernel size , 2017 .

[34]  Hassan Hammouri,et al.  Observer-based approach to fault detection and isolation for nonlinear systems , 1999, IEEE Trans. Autom. Control..

[35]  Deniz Erdoğmuş.  INFORMATION THEORETIC LEARNING: RENYI'S ENTROPY AND ITS APPLICATIONS TO ADAPTIVE SYSTEM TRAINING , 2002 .

[36]  M. Schatzman,et al.  Numerical Analysis: A Mathematical Introduction , 2002 .

[37]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..

[38]  Ghosheh Abed Hodtani,et al.  A new robust correntropy based Levenberg-Marquardt algorithm , 2016, 2016 Iran Workshop on Communication and Information Theory (IWCIT).

[39]  Ling Yu,et al.  Cyclic correntropy and its spectrum in frequency estimation in the presence of impulsive noise , 2016, Signal Process..

[40]  J. Príncipe,et al.  A family of minimum renyi's error entropy algorithm for information processing , 2007 .

[41]  Hak-Keung Lam,et al.  Observer-Based Fault Detection for Nonlinear Systems With Sensor Fault and Limited Communication Capacity , 2016, IEEE Transactions on Automatic Control.

[42]  José Carlos Príncipe,et al.  A closed form recursive solution for Maximum Correntropy training , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[43]  Ling Yu,et al.  A Time Delay Estimation Algorithm Based on the Weighted Correntropy Spectral Density , 2017, Circuits Syst. Signal Process..

[44]  Bo Li,et al.  Information Theoretic Subspace Clustering , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Azam Khalili,et al.  Steady-State Tracking Analysis of Adaptive Filter With Maximum Correntropy Criterion , 2017, Circuits Syst. Signal Process..

[46]  M. Fukushima,et al.  On the Rate of Convergence of the Levenberg-Marquardt Method , 2001 .

[47]  Nanning Zheng,et al.  A variable step-size adaptive algorithm under maximum correntropy criterion , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[48]  Deepak Nagaria,et al.  Design and Analysis of Cascaded LMS Adaptive Filters for Noise Cancellation , 2016, Circuits, Systems, and Signal Processing.

[49]  Howard M. Schwartz,et al.  Exponential convergence of the Kalman filter based parameter estimation algorithm , 2003 .

[50]  Raimon Jané,et al.  Correntropy-Based Spectral Characterization of Respiratory Patterns in Patients With Chronic Heart Failure , 2010, IEEE Transactions on Biomedical Engineering.