A New Information Theoretic Relation Between Minimum Error Entropy and Maximum Correntropy

The past decade has seen the rapid development of information theoretic learning and its applications in signal processing and machine learning. Specifically, minimum error entropy (MEE) and maximum correntropy criterion (MCC) have been widely studied in the literature. Although MEE and MCC are applied in many branches of knowledge and could outperform statistical criteria (such as mean square error), they have not been compared with each other from theoretical point of view. In some cases, MEE and MCC perform similarly to each other; however, under some conditions (e.g., in non-Gaussian environments), they act differently. This letter derives a new information theoretic relation between MEE and MCC, leading to better understanding of the theoretical differences, and illustrates the findings in a common example.

[1]  Deniz Erdoğmuş.  INFORMATION THEORETIC LEARNING: RENYI'S ENTROPY AND ITS APPLICATIONS TO ADAPTIVE SYSTEM TRAINING , 2002 .

[2]  Hongbin Zhang,et al.  Kernel Recursive Generalized Maximum Correntropy , 2017, IEEE Signal Processing Letters.

[3]  Shukai Duan,et al.  A Robust Diffusion Estimation Algorithm with Self-Adjusting Step-Size in WSNs , 2017, Sensors.

[4]  J. Príncipe,et al.  A family of minimum renyi's error entropy algorithm for information processing , 2007 .

[5]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..

[6]  Ghosheh Abed Hodtani,et al.  A new robust correntropy based Levenberg-Marquardt algorithm , 2016, 2016 Iran Workshop on Communication and Information Theory (IWCIT).

[7]  Hong-Jie Xing,et al.  Training extreme learning machine via regularized correntropy criterion , 2012, Neural Computing and Applications.

[8]  Ming Zhang,et al.  A new interpretation on the MMSE as a robust MEE criterion , 2010, Signal Process..

[9]  Junghui Chen,et al.  Correntropy Kernel Learning for Nonlinear System Identification with Outliers , 2014 .

[10]  Zongze Wu,et al.  Proportionate Minimum Error Entropy Algorithm for Sparse System Identification , 2015, Entropy.

[11]  Deniz Erdogmus,et al.  A mutual information extension to the matched filter , 2005, Signal Process..

[12]  Deniz Erdogmus,et al.  A minimum-error entropy criterion with self-adjusting step-size (MEE-SAS) , 2007, Signal Process..

[13]  Fei Wang,et al.  Maximum total correntropy adaptive filtering against heavy-tailed noises , 2017, Signal Process..

[14]  Nanning Zheng,et al.  Convergence of a Fixed-Point Algorithm under Maximum Correntropy Criterion , 2015, IEEE Signal Processing Letters.

[15]  Yu Zhang,et al.  Convergence of a Fixed-Point Minimum Error Entropy Algorithm , 2015, Entropy.

[16]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[17]  Jose C. Principe,et al.  Information Theoretic Learning - Renyi's Entropy and Kernel Perspectives , 2010, Information Theoretic Learning.

[18]  Ran He,et al.  Robust Principal Component Analysis Based on Maximum Correntropy Criterion , 2011, IEEE Transactions on Image Processing.

[19]  José Carlos Príncipe,et al.  Complex Correntropy: Probabilistic Interpretation and Application to Complex-Valued Data , 2017, IEEE Signal Processing Letters.

[20]  Weifeng Liu,et al.  Correntropy: Properties and Applications in Non-Gaussian Signal Processing , 2007, IEEE Transactions on Signal Processing.

[21]  Xiaojing Chen,et al.  Robust Cyclic MUSIC Algorithm for Finding Directions in Impulsive Noise Environment , 2017 .

[22]  Nanning Zheng,et al.  Kernel minimum error entropy algorithm , 2013, Neurocomputing.

[23]  Ghosheh Abed Hodtani,et al.  A new robust fixed-point algorithm and its convergence analysis , 2017 .

[24]  Jian-Wu Xu,et al.  Nonlinear Signal Processing Based on Reproducing Kernel Hilbert Space , 2010 .

[25]  Zongze Wu,et al.  Adaptive convex combination filter under minimum error entropy criterion , 2016, 2016 IEEE International Conference on Digital Signal Processing (DSP).

[26]  Ghosheh Abed Hodtani,et al.  Generalized maximum correntropy detector for non‐Gaussian environments , 2018 .

[27]  Peter E. William,et al.  Error Entropy and Mean Square Error Minimization for Lossless Image Compression , 2006, 2006 International Conference on Image Processing.

[28]  Zeng-qi Sun,et al.  Information Theoretic Interpretation of Error Criteria , 2009 .

[29]  P. P. Pokharel,et al.  Error Entropy, Correntropy and M-Estimation , 2006, 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing.

[30]  Liang Wang,et al.  Robust Recognition via Information Theoretic Learning , 2014, SpringerBriefs in Computer Science.