Multiple Random Empirical Kernel Learning with Margin Reinforcement for imbalance problems

Abstract Imbalance problems arise in real-world applications when the number of negative samples far exceeds the number of positive samples, such as medical data. When solving the classification of imbalance problems, the samples located near the decision hyperplane contribute more to the decision hyperplane, and the samples far from the decision hyperplane contribute less to the decision hyperplane. So we can consider giving higher weights to the samples near the decision hyperplane, but they are sensitive to noise, and too much emphasis on them may lead to unstable performance. This paper proposes a Margin Reinforcement (MR) method to overcome the above dilemma. Because the imbalance problem is a cost-sensitive problem, MR gives positive samples a uniform high weight to improve the misclassification cost of the positive sample. For negative samples, according to their entropy, samples away from the decision surface and noise samples mixed in the positive samples are given a smaller weight, in order to improve the efficiency and robustness of the algorithm. Therefore, MR can emphasize the importance of samples located in overlapping regions of positive and negative classes and ignore the effects of noise samples to produce superior performance. Multiple Random Empirical Kernel Learning (MREKL) has proven to be effective and efficient in dealing with balance problems. In order to improve the performance of MREKL on imbalanced datasets, MR is introduced into MREKL to propose a novel Multiple Random Empirical Kernel Learning with Margin Reinforcement (MREKL-MR). MREKL-MR efficiently map the samples into low-dimensional feature spaces, then utilizes the MR approach to reenforce the importance of margin samples and decrease the effects of noise samples. Experimental results on 28 imbalanced datasets indicate that MREKL-MR is superior to comparison algorithms. Finally, the effectiveness of MREKL-MR in dealing with imbalance problems is verified on the Heart Failure dataset.

[1]  Chuanhou Gao,et al.  Design of a multiple kernel learning algorithm for LS-SVM by convex programming , 2011, Neural Networks.

[2]  Christophe Mues,et al.  An experimental comparison of classification algorithms for imbalanced credit scoring data sets , 2012, Expert Syst. Appl..

[3]  Lale Özbakir,et al.  A cost-sensitive classification algorithm: BEE-Miner , 2016, Knowl. Based Syst..

[4]  Mayank Sharma,et al.  Sparsity in function and derivative approximation via the empirical feature space , 2020, Inf. Sci..

[5]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[6]  Fernando Bação,et al.  Oversampling for Imbalanced Learning Based on K-Means and SMOTE , 2017, Inf. Sci..

[7]  Antônio de Pádua Braga,et al.  Novel Cost-Sensitive Approach to Improve the Multilayer Perceptron Performance on Imbalanced Data , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Zhe Wang,et al.  Cascade interpolation learning with double subspaces and confidence disturbance for imbalanced problems , 2019, Neural Networks.

[9]  Dacheng Tao,et al.  Eigenfunction-Based Multitask Learning in a Reproducing Kernel Hilbert Space , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Sheng-De Wang,et al.  Fuzzy support vector machines , 2002, IEEE Trans. Neural Networks.

[11]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[12]  Zhe Wang,et al.  Geometric Structural Ensemble Learning for Imbalanced Problems , 2020, IEEE Transactions on Cybernetics.

[13]  Dongdong Li,et al.  Collaborative and geometric multi-kernel learning for multi-class classification , 2020, Pattern Recognit..

[14]  Francisco Herrera,et al.  Imbalance: Oversampling algorithms for imbalanced classification in R , 2018, Knowl. Based Syst..

[15]  Francisco Herrera,et al.  EUSBoost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling , 2013, Pattern Recognit..

[16]  Opim Salim Sitompul,et al.  Biased support vector machine and weighted-smote in handling class imbalance problem , 2018 .

[17]  Ethem Alpaydin,et al.  Regularizing multiple kernel learning using response surface methodology , 2011, Pattern Recognit..

[18]  Julio López,et al.  Imbalanced data classification using second-order cone programming support vector machines , 2014, Pattern Recognit..

[19]  Hongyuan Zha,et al.  Entropy-based fuzzy support vector machine for imbalanced datasets , 2017, Knowl. Based Syst..

[20]  Alicia Pérez,et al.  The class imbalance problem detecting adverse drug reactions in electronic health records , 2018, Health Informatics J..

[21]  Michail G. Lagoudakis,et al.  Least-Squares Policy Iteration , 2003, J. Mach. Learn. Res..

[22]  Francisco Herrera,et al.  A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[23]  Hongyuan Zha,et al.  MREKLM: A fast multiple empirical kernel learning machine , 2017, Pattern Recognit..

[24]  John Langford,et al.  Active Learning for Cost-Sensitive Classification , 2017, ICML.

[25]  Junjun Jiang,et al.  LMR: Learning a Two-Class Classifier for Mismatch Removal , 2019, IEEE Transactions on Image Processing.

[26]  María José del Jesús,et al.  KEEL: a software tool to assess evolutionary algorithms for data mining problems , 2008, Soft Comput..

[27]  Xiaowei Yang,et al.  A Kernel Fuzzy c-Means Clustering-Based Fuzzy Support Vector Machine Algorithm for Classification Problems With Outliers or Noises , 2011, IEEE Transactions on Fuzzy Systems.

[28]  Pedro Antonio Gutiérrez,et al.  Oversampling the Minority Class in the Feature Space , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[29]  Hongyuan Zha,et al.  Boundary-Eliminated Pseudoinverse Linear Discriminant for Imbalanced Problems , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Xun Liang,et al.  Feature space versus empirical kernel map and row kernel space in SVMs , 2010, Neural Computing and Applications.

[31]  Edward R. Dougherty,et al.  Is cross-validation valid for small-sample microarray classification? , 2004, Bioinform..

[32]  Charles X. Ling,et al.  Using AUC and accuracy in evaluating learning algorithms , 2005, IEEE Transactions on Knowledge and Data Engineering.

[33]  Feng Hao,et al.  Cryptanalysis of a Chaotic Image Encryption Algorithm Based on Information Entropy , 2018, IEEE Access.

[34]  Dinggang Shen,et al.  Structured sparsity regularized multiple kernel learning for Alzheimer's disease diagnosis , 2019, Pattern Recognit..

[35]  Tom Diethe,et al.  Online Learning with (Multiple) Kernels: A Review , 2013, Neural Computation.

[36]  Atsuto Maki,et al.  A systematic study of the class imbalance problem in convolutional neural networks , 2017, Neural Networks.

[37]  José Salvador Sánchez,et al.  On the k-NN performance in a challenging scenario of imbalance and overlapping , 2008, Pattern Analysis and Applications.

[38]  Hector Budman,et al.  A hybrid clustering approach for multivariate time series - A case study applied to failure analysis in a gas turbine. , 2017, ISA transactions.

[39]  Javier Pérez-Rodríguez,et al.  OligoIS: Scalable Instance Selection for Class-Imbalanced Data Sets , 2013, IEEE Transactions on Cybernetics.

[40]  Ivor W. Tsang,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1 Soft Margin Multiple Kernel Learning , 2022 .

[41]  Adi Ben-Israel,et al.  Generalized inverses: theory and applications , 1974 .