Training set selection and swarm intelligence for enhanced integration in multiple classifier systems

Abstract Multiple classifier systems (MCSs) constitute one of the most competitive paradigms for obtaining more accurate predictions in the field of machine learning. Systems of this type should be designed efficiently in all of their stages, from data preprocessing to multioutput decision fusion. In this article, we present a framework for utilizing the power of instance selection methods and the search capabilities of swarm intelligence to train learning models and to aggregate their decisions. The process consists of three steps: First, the essence of the complete training data set is captured in a reduced set via the application of intelligent data sampling. Second, the reduced set is used to train a group of heterogeneous classifiers using bagging and distance-based feature sampling. Finally, swarm intelligence techniques are applied to identify a pattern among multiple decisions to enhance the fusion process by assigning class-specific weights for each classifier. The proposed methodology yielded competitive results in experiments that were conducted on 25 benchmark datasets. The Matthews correlation coefficient (MCC) is regarded as the objective to be maximized by various nature-inspired metaheuristics, which include the moth-flame optimization algorithm (MFO), the grey wolf optimizer (GWO) and the whale optimization algorithm (WOA).

[1]  Andrew Lewis,et al.  The Whale Optimization Algorithm , 2016, Adv. Eng. Softw..

[2]  Melanie Mitchell,et al.  An introduction to genetic algorithms , 1996 .

[3]  Bartosz Krawczyk,et al.  Ensemble of Extreme Learning Machines with trained classifier combination and statistical features for hyperspectral data , 2018, Neurocomputing.

[4]  Asif Ekbal,et al.  Combining feature selection and classifier ensemble using a multiobjective simulated annealing approach: application to named entity recognition , 2012, Soft Computing.

[5]  Luisa Micó,et al.  Comparison of Classifier Fusion Methods for Classification in Pattern Recognition Tasks , 2006, SSPR/SPR.

[6]  Álvaro García-Martín,et al.  People detection in surveillance: classification and evaluation , 2015, IET Comput. Vis..

[7]  Juan José Rodríguez Diez,et al.  A weighted voting framework for classifiers ensembles , 2012, Knowledge and Information Systems.

[8]  Gang Niu,et al.  Multi-agent decision fusion for motor fault diagnosis , 2007 .

[9]  Jason J. Jung,et al.  Social big data: Recent achievements and new challenges , 2015, Information Fusion.

[10]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[11]  Francisco Herrera,et al.  Data Preprocessing in Data Mining , 2014, Intelligent Systems Reference Library.

[12]  F. Wilcoxon,et al.  Individual comparisons of grouped data by ranking methods. , 1946, Journal of economic entomology.

[13]  B. Matthews.  Comparison of the predicted and observed secondary structure of T4 phage lysozyme. , 1975, Biochimica et biophysica acta.

[14]  Lior Rokach,et al.  Ensemble learning: A survey , 2018, WIREs Data Mining Knowl. Discov..

[15]  Zhi-Hua Zhou,et al.  Ensemble Methods: Foundations and Algorithms , 2012 .

[16]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[17]  Aboul Ella Hassanien,et al.  Swarm Intelligence: Principles, Advances, and Applications , 2015 .

[18]  Nojun Kwak,et al.  Feature extraction for classification problems and its application to face recognition , 2008, Pattern Recognit..

[19]  Marco Dorigo.  Ant colony optimization , 2004, Scholarpedia.

[20]  Andrew Lewis,et al.  Grey Wolf Optimizer , 2014, Adv. Eng. Softw..

[21]  Seyed Mohammad Mirjalili,et al.  Moth-flame optimization algorithm: A novel nature-inspired heuristic paradigm , 2015, Knowl. Based Syst..

[22]  Emilio Corchado,et al.  A survey of multiple classifier systems as hybrid systems , 2014, Inf. Fusion.

[23]  Miguel Toro,et al.  Finding representative patterns with ordered projections , 2003, Pattern Recognit..

[24]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[25]  Cagatay CATAL,et al.  A sentiment classification model based on multiple classifiers , 2017, Appl. Soft Comput..

[26]  Dervis Karaboga,et al.  A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm , 2007, J. Glob. Optim..

[27]  Julian Fiérrez,et al.  Multiple classifiers in biometrics. part 1: Fundamentals and review , 2018, Inf. Fusion.

[28]  Asif Ekbal,et al.  Weighted Vote-Based Classifier Ensemble for Named Entity Recognition: A Genetic Algorithm-Based Approach , 2011, TALIP.

[29]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[30]  Riyaz Sikora,et al.  A Modified Stacking Ensemble Machine Learning Algorithm Using Genetic Algorithms , 2014, Journal of International Technology and Information Management.

[31]  Luca Maria Gambardella,et al.  A survey on metaheuristics for stochastic combinatorial optimization , 2009, Natural Computing.

[32]  Xin-She Yang,et al.  A New Metaheuristic Bat-Inspired Algorithm , 2010, NICSO.

[33]  Jing Qiu,et al.  Dynamic weighted ensemble classification for credit scoring using Markov Chain , 2018, Applied Intelligence.

[34]  Robert Sabourin,et al.  From dynamic classifier selection to dynamic ensemble selection , 2008, Pattern Recognit..

[35]  J. R. Parker,et al.  Rank and response combination from confusion matrix data , 2001, Inf. Fusion.

[36]  Francisco Herrera,et al.  Prototype Selection for Nearest Neighbor Classification: Taxonomy and Empirical Study , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[38]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[39]  Michal Wozniak.  Experiments on Linear Combiners , 2008, Information Technologies in Biomedicine.

[40]  Francisco Herrera,et al.  A memetic algorithm for evolutionary prototype selection: A scaling up approach , 2008, Pattern Recognit..

[41]  Gerald Schaefer,et al.  Cost-sensitive decision tree ensembles for effective imbalanced classification , 2014, Appl. Soft Comput..

[42]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[43]  Mamun Bin Ibne Reaz,et al.  A novel SVM-kNN-PSO ensemble method for intrusion detection system , 2016, Appl. Soft Comput..

[44]  Cesare Furlanello,et al.  A Comparison of MCC and CEN Error Measures in Multi-Class Prediction , 2010, PloS one.

[45]  Francisco Herrera,et al.  Evolutionary stratified training set selection for extracting classification rules with trade off precision-interpretability , 2007, Data Knowl. Eng..

[46]  Jesús Alcalá-Fdez,et al.  KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework , 2011, J. Multiple Valued Log. Soft Comput..

[47]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[48]  Aytug Onan,et al.  A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification , 2016, Expert Syst. Appl..