Cost-sensitive probability for weighted voting in an ensemble model for multi-class classification problems

Ensemble learning is an algorithm that utilizes various types of classification models. This algorithm can enhance the prediction efficiency of component models. However, the efficiency of combining models typically depends on the diversity and accuracy of the predicted results of ensemble models. However, the problem of multi-class data is still encountered. In the proposed approach, cost-sensitive learning was implemented to evaluate the prediction accuracy for each class, which was used to construct a cost-sensitivity matrix of the true positive (TP) rate. This TP rate can be used as a weight value and combined with a probability value to drive ensemble learning for a specified class. We proposed an ensemble model, which was a type of heterogenous model, namely, a combination of various individual classification models (support vector machine, Bayes, K-nearest neighbour, naive Bayes, decision tree, and multi-layer perceptron) in experiments on 3-, 4-, 5- and 6-classifier models. The efficiencies of the propose models were compared to those of the individual classifier model and homogenous models (Adaboost, bagging, stacking, voting, random forest, and random subspaces) with various multi-class data sets. The experimental results demonstrate that the cost-sensitive probability for the weighted voting ensemble model that was derived from 3 models provided the most accurate results for the dataset in multi-class prediction. The objective of this study was to increase the efficiency of predicting classification results in multi-class classification tasks and to improve the classification results.

[1]  Shraddha S. More,et al.  Trust-based Voting Method for Efficient Malware Detection , 2016 .

[2]  Fernando Benites,et al.  Multi-label classification and extracting predicted class hierarchies , 2011, Pattern Recognit..

[3]  Panos M. Pardalos,et al.  A novel perspective on multiclass classification: Regular simplex support vector machine , 2019, Inf. Sci..

[4]  Leonardo Vanneschi,et al.  Multidimensional genetic programming for multiclass classification , 2019, Swarm Evol. Comput..

[5]  Marek Kurzynski,et al.  Weighting scheme for a pairwise multi-label classifier based on the fuzzy confusion matrix , 2018, Pattern Recognit. Lett..

[6]  Leonardo Ramos Rodrigues,et al.  Building selective ensembles of Randomization Based Neural Networks with the successive projections algorithm , 2017, Appl. Soft Comput..

[7]  Juan José Rodríguez Diez,et al.  A weighted voting framework for classifiers ensembles , 2012, Knowledge and Information Systems.

[8]  Enrique Herrera-Viedma,et al.  Large-Scale decision-making: Characterization, taxonomy, challenges and future directions from an Artificial Intelligence and applications perspective , 2020, Inf. Fusion.

[9]  Randy C. Paffenroth,et al.  Random Forests for mapping and analysis of microkinetics models , 2018, Comput. Chem. Eng..

[10]  Yuming Zhou,et al.  A novel ensemble method for classifying imbalanced data , 2015, Pattern Recognit..

[11]  Liberios Vokorokos,et al.  Ensemble feature selection using election methods and ranker clustering , 2019, Inf. Sci..

[12]  M. E. Banihabib,et al.  A decision-making model for flood warning system based on ensemble forecasts , 2019, Journal of Hydrology.

[13]  Chen Lu,et al.  Fault diagnosis for rotary machinery with selective ensemble neural networks , 2017, Mechanical Systems and Signal Processing.

[14]  Xiwang Li,et al.  Using an ensemble machine learning methodology-Bagging to predict occupants’ thermal comfort in buildings , 2018, Energy and Buildings.

[15]  Ran Wang,et al.  An analysis on the relationship between uncertainty and misclassification rate of classifiers , 2020, Inf. Sci..

[16]  Tinghuai Ma,et al.  Detecting potential labeling errors for bioinformatics by multiple voting , 2014, Knowl. Based Syst..

[17]  Xiong Luo,et al.  Improving Classification Performance through an Advanced Ensemble Based Heterogeneous Extreme Learning Machines , 2017, Comput. Intell. Neurosci..

[18]  Raj Kamal,et al.  A hybrid ensemble for classification in multiclass datasets: An application to oilseed disease dataset , 2016, Comput. Electron. Agric..

[19]  Dini Rosdini,et al.  Decision making biased: How visual illusion, mood, and information presentation plays a role , 2020 .

[20]  Ladislav A. Novak,et al.  Hardware acceleration of homogeneous and heterogeneous ensemble classifiers , 2015, Microprocess. Microsystems.

[21]  A. Enk,et al.  Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks. , 2019, European journal of cancer.

[22]  Yang Hui,et al.  Prediction of component content in rare earth extraction process based on ESNs-Adaboost , 2018 .

[23]  Hongren Gong,et al.  Use of random forests regression for predicting IRI of asphalt pavements , 2018, Construction and Building Materials.

[24]  Mohsen Mesgarpour,et al.  Ensemble Risk Model of Emergency Admissions (ERMER) , 2017, Int. J. Medical Informatics.

[25]  Guoping Qiu,et al.  Random Forest for Label Ranking , 2016, Expert Syst. Appl..

[26]  Mucahid Barstugan,et al.  The effect of dictionary learning on weight update of AdaBoost and ECG classification , 2018, J. King Saud Univ. Comput. Inf. Sci..

[27]  Hongshik Ahn,et al.  A weight-adjusted voting algorithm for ensembles of classifiers , 2011 .

[28]  Liangxiao Jiang,et al.  Class-specific attribute weighted naive Bayes , 2019, Pattern Recognit..

[29]  Niall Rooney,et al.  A weighted combination of stacking and dynamic integration , 2007, Pattern Recognit..

[30]  Hussein Samma,et al.  Optimized Two-Stage Ensemble Model for Mammography Mass Recognition , 2020 .

[31]  José Ramón Quevedo,et al.  Dynamic ensemble selection for quantification tasks , 2019, Inf. Fusion.

[32]  César Ferri,et al.  Improving Performance of Multiclass Classification by Inducing Class Hierarchies , 2017, ICCS.

[33]  Irena Koprinska,et al.  Multi-step forecasting for big data time series based on ensemble learning , 2019, Knowl. Based Syst..

[34]  Maryam Sabzevari,et al.  Vote-boosting ensembles , 2016, Pattern Recognit..

[35]  Concha Bielza,et al.  Learning tractable Bayesian networks in the space of elimination orders , 2019, Artif. Intell..

[36]  Asif Ekbal,et al.  Combining multiple classifiers using vote based classifier ensemble technique for named entity recognition , 2013, Data Knowl. Eng..

[37]  Mustafa ElNainay,et al.  Software bug prediction using weighted majority voting techniques , 2018, Alexandria Engineering Journal.

[38]  C. V. Jawahar,et al.  Improving multiclass classification by deep networks using DAGSVM and Triplet Loss , 2018, Pattern Recognit. Lett..

[39]  Tri Dev Acharya,et al.  Landslide susceptibility mapping using J48 Decision Tree with AdaBoost, Bagging and Rotation Forest ensembles in the Guangchang area (China) , 2018 .

[40]  Dazhong Wu,et al.  An ensemble learning-based prognostic approach with degradation-dependent weights for remaining useful life prediction , 2017, Reliab. Eng. Syst. Saf..

[41]  Dianhui Wang,et al.  Fast decorrelated neural network ensembles with random weights , 2014, Inf. Sci..

[42]  Graeme Garner,et al.  Ensemble modelling and structured decision-making to support Emergency Disease Management. , 2017, Preventive veterinary medicine.

[43]  Gholam Ali Montazer,et al.  Improving multiclass classification using neighborhood search in error correcting output codes , 2017, Pattern Recognit. Lett..

[44]  Hamido Fujita,et al.  Imbalanced enterprise credit evaluation with DTE-SBD: Decision tree ensemble based on SMOTE and bagging with differentiated sampling rates , 2018, Inf. Sci..

[45]  Zied Elouedi,et al.  Decision tree classifiers for evidential attribute values and class labels , 2019, Fuzzy Sets Syst..

[46]  Guo Cao,et al.  A novel ensemble method for k-nearest neighbor , 2019, Pattern Recognit..

[47]  Yaochu Jin,et al.  Stacking-based ensemble learning of decision trees for interpretable prostate cancer detection , 2019, Appl. Soft Comput..

[48]  Jie Xu,et al.  Multi-model ensemble with rich spatial information for object detection , 2020, Pattern Recognit..

[49]  T. Bakhshpoori,et al.  Improving the prediction of ground motion parameters based on an efficient bagging ensemble model of M5′ and CART algorithms , 2018, Appl. Soft Comput..

[50]  Zeyu Wang,et al.  A review of artificial intelligence based building energy use prediction: Contrasting the capabilities of single and ensemble prediction models , 2017 .

[51]  Sam Kwong,et al.  Class-specific soft voting based multiple extreme learning machines ensemble , 2015, Neurocomputing.

[52]  Wei Sun,et al.  A stacking ensemble learning framework for annual river ice breakup dates , 2018, Journal of Hydrology.

[53]  Kin Keung Lai,et al.  A decomposition clustering ensemble learning approach for forecasting foreign exchange rates , 2019, Journal of Management Science and Engineering.

[54]  Hamit Erdal,et al.  Bagging ensemble models for bank profitability: An emprical research on Turkish development and investment banks , 2016, Appl. Soft Comput..

[55]  Mohsen Mesgarpour,et al.  Corrigendum to "Ensemble Risk Model of Emergency Admissions (ERMER)"[Int. J. Med. Inf. 103 (2017) 65-77] , 2017, Int. J. Medical Informatics.

[56]  Zhong Liu,et al.  A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM , 2017, Comput. Intell. Neurosci..

[57]  Zhe Zhu,et al.  Mapping forest change using stacked generalization: An ensemble approach , 2018 .

[58]  J. Weller,et al.  Dimensions of decision-making: An evidence-based classification of heuristics and biases , 2019, Personality and Individual Differences.

[59]  Zhiqiang Ge,et al.  Ensemble semi-supervised Fisher discriminant analysis model for fault classification in industrial processes. , 2019, ISA transactions.

[60]  Tim Barden,et al.  Automated defect detection for Fluorescent Penetrant Inspection using Random Forest , 2019, NDT & E International.

[61]  Michael G. Wheaton,et al.  Maximizing decision-making style and hoarding disorder symptoms. , 2020, Comprehensive psychiatry.

[62]  Maria do Carmo Nicoletti,et al.  An iterative boosting-based ensemble for streaming data classification , 2019, Inf. Fusion.

[63]  Bo Chen,et al.  Weighted classifier ensemble based on quadratic form , 2015, Pattern Recognit..

[64]  Sungzoon Cho,et al.  Multi-class classification via heterogeneous ensemble of one-class classifiers , 2015, Eng. Appl. Artif. Intell..

[65]  Rui Liu,et al.  Self-adaptive cost weights-based support vector machine cost-sensitive ensemble for imbalanced data classification , 2019, Inf. Sci..

[66]  Lin Lin,et al.  Random forests-based extreme learning machine ensemble for multi-regime time series prediction , 2017, Expert Syst. Appl..

[67]  Liu Xiao,et al.  Adapted ensemble classification algorithm based on multiple classifier system and feature selection for classifying multi-class imbalanced data , 2016 .

[68]  Rudiyanto Gunawan,et al.  Bioprocess optimization under uncertainty using ensemble modeling. , 2017, Journal of biotechnology.

[69]  Shankru Guggari,et al.  Non-sequential partitioning approaches to decision tree classifier , 2018, Future Computing and Informatics Journal.

[70]  Fang Liu,et al.  Random subspace based ensemble sparse representation , 2018, Pattern Recognit..

[71]  Prakash P. Shenoy,et al.  A bias-variance based heuristic for constructing a hybrid logistic regression-naïve Bayes model for classification , 2020, Int. J. Approx. Reason..

[72]  Brian Mac Namee,et al.  Kalman Filter-based Heuristic Ensemble (KFHE): A new perspective on multi-class ensemble classification using Kalman filters , 2018, Inf. Sci..

[73]  Hongqiu Zhu,et al.  Simultaneous determination of trace amounts of copper and cobalt in high concentration zinc solution using UV–vis spectrometry and Adaboost , 2019, Optik.

[74]  S. van Dulmen,et al.  Building ground for didactics in a patient decision aid for hip osteoarthritis. Exploring patient-related barriers and facilitators towards shared decision-making. , 2020, Patient education and counseling.

[75]  Aytug Onan,et al.  A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification , 2016, Expert Syst. Appl..

[76]  Xiaobo Liu,et al.  Hierarchical ensemble of Extreme Learning Machine , 2018, Pattern Recognit. Lett..

[77]  Ioannis A. Kakadiaris,et al.  Hierarchical Multi-label Classification using Fully Associative Ensemble Learning , 2017, Pattern Recognit..

[78]  Hossein Shamsi,et al.  Analog programmable neuron and case study on VLSI implementation of Multi-Layer Perceptron (MLP) , 2019, Microelectron. J..

[79]  Laurence Alison,et al.  Military maximizers: Examining the effect of individual differences in maximization on military decision-making , 2020 .

[80]  José Salvador Sánchez,et al.  A bias correction function for classification performance assessment in two-class imbalanced problems , 2014, Knowl. Based Syst..

[81]  Xinli Yang,et al.  TLEL: A two-layer ensemble learning approach for just-in-time defect prediction , 2017, Inf. Softw. Technol..