Cost-sensitive Fuzzy Multiple Kernel Learning for imbalanced problem

Abstract Multiple Kernel Learning (MKL) improves the classification accuracy by exploring different formulations of data. However, its classification performance is still unsatisfactory in imbalanced problems. To address this issue, we take the characteristics of imbalanced data into account by introducing the fuzzy memberships. In our work, the fuzzy memberships are determined by both the entropies of samples and the cost for each class so as to make different samples have different contributions to the decision boundary. Thus the newly proposed method can result in more favorable classification performances on imbalanced datasets. Further, we introduce the fuzzy memberships into existing MKL to form a new algorithm Cost-sensitive Fuzzy Multiple Kernel Learning named CFMKL in short. Experimental results validate the great effectiveness of the proposed CFMKL on synthetic, real-world binary and multi-class imbalanced datasets. The major contributions of this paper are as follows. Firstly, extending the MKL to handle the imbalanced problems for the first time. Secondly, generating a new fuzzy memberships function by both the entropy and the cost-sensitive. Thirdly, proposing a new algorithm named CFMKL for imbalanced problems and validating its effectiveness.

[1]  Theoden Netoff,et al.  Seizure prediction with spectral power of EEG using cost‐sensitive support vector machines , 2011, Epilepsia.

[2]  Sheng-De Wang,et al.  Fuzzy support vector machines , 2002, IEEE Trans. Neural Networks.

[3]  Zahra Nazari,et al.  Density Based Support Vector Machines for Classification , 2015 .

[4]  Lei Wang,et al.  An Efficient Approach to Integrating Radius Information into Multiple Kernel Learning , 2013, IEEE Transactions on Cybernetics.

[5]  Alex Smola,et al.  Kernel methods in machine learning , 2007, math/0701907.

[6]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[7]  Zenglin Xu,et al.  Auto-weighted multi-view clustering via kernelized graph learning , 2019, Pattern Recognit..

[8]  Changming Zhu,et al.  Entropy-based matrix learning machine for imbalanced data sets , 2017, Pattern Recognit. Lett..

[9]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[10]  María José del Jesús,et al.  KEEL: a software tool to assess evolutionary algorithms for data mining problems , 2008, Soft Comput..

[11]  Songcan Chen,et al.  Multi-view kernel machine on single-view data , 2009, Neurocomputing.

[12]  Erik Cambria,et al.  Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis , 2017, Neurocomputing.

[13]  Ivor W. Tsang,et al.  Generalized Multiple Kernel Learning With Data-Dependent Priors , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[14]  Fernando Bação,et al.  Density Based Fuzzy Membership Functions in the Context of Geocomputation , 2007, International Conference on Computational Science.

[15]  Fabio Aiolli,et al.  EasyMKL: a scalable multiple kernel learning algorithm , 2015, Neurocomputing.

[16]  Jerzy Stefanowski,et al.  Neighbourhood sampling in bagging for imbalanced data , 2015, Neurocomputing.

[17]  Yung-Yu Chuang,et al.  Multiple Kernel Fuzzy Clustering , 2012, IEEE Transactions on Fuzzy Systems.

[18]  Alessandro Laio,et al.  Clustering by fast search and find of density peaks , 2014, Science.

[19]  Zhi-Hua Zhou,et al.  ON MULTI‐CLASS COST‐SENSITIVE LEARNING , 2006, Comput. Intell..

[20]  Nathalie Japkowicz,et al.  The class imbalance problem: A systematic study , 2002, Intell. Data Anal..

[21]  Steven C. H. Hoi,et al.  Cost-Sensitive Online Classification , 2012, 2012 IEEE 12th International Conference on Data Mining.

[22]  Dinggang Shen,et al.  Late Fusion Incomplete Multi-View Clustering , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Jocelyn Chanussot,et al.  Multiple Kernel Learning for Hyperspectral Image Classification: A Review , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Simon Fong,et al.  An Application of Oversampling, Undersampling, Bagging and Boosting in Handling Imbalanced Datasets , 2013, DaEng.

[25]  MengChu Zhou,et al.  Fuzzy support vector machine based on density with dual membership , 2009, 2009 International Conference on Machine Learning and Cybernetics.

[26]  Lei Wang,et al.  Multiple kernel extreme learning machine , 2015, Neurocomputing.

[27]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[28]  Aristidis Likas,et al.  Kernel-Based Weighted Multi-view Clustering , 2012, 2012 IEEE 12th International Conference on Data Mining.

[29]  Rong Jin,et al.  Multiple Kernel Learning for Visual Object Recognition: A Review , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Bartosz Krawczyk,et al.  Clustering-based ensembles for one-class classification , 2014, Inf. Sci..

[31]  Zenglin Xu,et al.  Simple and Efficient Multiple Kernel Learning by Group Lasso , 2010, ICML.

[32]  Virginia R. de Sa,et al.  Learning Classification with Unlabeled Data , 1993, NIPS.

[33]  Jacek M. Łȩski,et al.  Ho--Kashyap classifier with generalization control , 2003 .

[34]  Mehryar Mohri,et al.  Learning Non-Linear Combinations of Kernels , 2009, NIPS.

[35]  Francisco Herrera,et al.  A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[36]  Shehroz S. Khan,et al.  One-class classification: taxonomy of study and review of techniques , 2013, The Knowledge Engineering Review.

[37]  Vasile Palade,et al.  FSVM-CIL: Fuzzy Support Vector Machines for Class Imbalance Learning , 2010, IEEE Transactions on Fuzzy Systems.

[38]  Andrew K. C. Wong,et al.  Classification of Imbalanced Data: a Review , 2009, Int. J. Pattern Recognit. Artif. Intell..

[39]  Lance Chun Che Fung,et al.  Emotion Classification from Electroencephalogram Using Fuzzy Support Vector Machine , 2017, ICONIP.

[40]  Francisco Herrera,et al.  An insight into imbalanced Big Data classification: outcomes and challenges , 2017 .

[41]  Jing Cheng,et al.  Affective detection based on an imbalanced fuzzy support vector machine , 2015, Biomed. Signal Process. Control..

[42]  Ivor W. Tsang,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1 Soft Margin Multiple Kernel Learning , 2022 .

[43]  Zhi-Hua Zhou,et al.  Large Margin Distribution Learning with Cost Interval and Unlabeled Data , 2016, IEEE Transactions on Knowledge and Data Engineering.

[44]  Anderson Rocha,et al.  Multiclass From Binary: Expanding One-Versus-All, One-Versus-One and ECOC-Based Approaches , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[45]  David G. Stork,et al.  Pattern Classification , 1973 .

[46]  Salvatore J. Stolfo,et al.  Cost-based modeling for fraud and intrusion detection: results from the JAM project , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[47]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[48]  Jing Yang,et al.  An Improved Random Forest Algorithm for Class-Imbalanced Data Classification and its Application in PAD Risk Factors Analysis , 2013 .

[49]  Songcan Chen,et al.  A novel multi-view learning developed from single-view patterns , 2011, Pattern Recognit..

[50]  Mohammad H. Mahoor,et al.  Human activity recognition using multi-features and multiple kernel learning , 2014, Pattern Recognit..

[51]  Dimitris Kanellopoulos,et al.  Handling imbalanced datasets: A review , 2006 .

[52]  Jingjing Tang,et al.  A multi-kernel framework with nonparallel support vector machine , 2017, Neurocomputing.

[53]  Zhe Wang,et al.  Gravitational fixed radius nearest neighbor for imbalanced problem , 2015, Knowl. Based Syst..

[54]  Rama Chellappa,et al.  Multiple Kernel Learning for Sparse Representation-Based Classification , 2014, IEEE Transactions on Image Processing.

[55]  Charles X. Ling,et al.  Using AUC and accuracy in evaluating learning algorithms , 2005, IEEE Transactions on Knowledge and Data Engineering.

[56]  Bartosz Krawczyk,et al.  Learning from imbalanced data: open challenges and future directions , 2016, Progress in Artificial Intelligence.

[57]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[58]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[59]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[60]  Li Zhang,et al.  Hybrid decision tree and naïve Bayes classifiers for multi-class classification tasks , 2014, Expert Syst. Appl..

[61]  Zhang Yi,et al.  Fuzzy SVM with a new fuzzy membership function , 2006, Neural Computing & Applications.

[62]  Ivor W. Tsang,et al.  A Feature Selection Method for Multivariate Performance Measures , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  Xin Yao,et al.  MWMOTE--Majority Weighted Minority Oversampling Technique for Imbalanced Data Set Learning , 2014 .

[64]  Jian Gao,et al.  A new sampling method for classifying imbalanced data based on support vector machine ensemble , 2016, Neurocomputing.

[65]  Vladimir Koltchinskii,et al.  Rademacher penalties and structural risk minimization , 2001, IEEE Trans. Inf. Theory.

[66]  Zhi-Hua Zhou,et al.  One-Pass AUC Optimization , 2013, ICML.

[67]  Michael I. Jordan,et al.  Multiple kernel learning, conic duality, and the SMO algorithm , 2004, ICML.

[68]  Lorenzo Bruzzone,et al.  Multiple Kernel Learning for Remote Sensing Image Classification , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[69]  David G. Lowe,et al.  Scalable Nearest Neighbor Algorithms for High Dimensional Data , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Kin Keung Lai,et al.  A new fuzzy support vector machine to evaluate credit risk , 2005, IEEE Transactions on Fuzzy Systems.

[71]  Hongyuan Zha,et al.  Entropy-based fuzzy support vector machine for imbalanced datasets , 2017, Knowl. Based Syst..

[72]  James C. Bezdek,et al.  On cluster validity for the fuzzy c-means model , 1995, IEEE Trans. Fuzzy Syst..