Stability investigation of multi-objective heuristic ensemble classifiers

Stability analysis of heuristic ensemble classifiers, which are designed by using heuristic methods, is a significant topic due to the stochastic nature of heuristic algorithms. Considering the importance of this issue, the novelty of this paper is stability analysis of heuristic ensemble classifiers. So, in this paper, at first, two multi-objective heuristic ensemble classifiers by using a new multi-objective heuristic approach called multi-objective inclined planes optimization (MOIPO) algorithm and a conventional one called multi-objective particle swarm optimization (MOPSO) algorithm are designed and then, two-level factorial designs, as a statistical approach, are applied to investigate the stability of the best ensemble classifier from two designed ensemble classifiers for the first time; for this purpose, the effects of three structural parameters of winner algorithm i.e. inflation rate, leader selection pressure and deletion selection pressure on the performance of designed heuristic ensemble classifier for three datasets as a representative of simple data, overlapped data and data with huge number of features are investigated. Extensive experimental and comparative results on different kinds of benchmarks with nonlinear, overlapping class boundaries and different feature space dimensions not only show the supremacy of MOIPO for designing ensemble classifiers but also the important parameters and important interactions for each objective function.

[1]  Noel E. Sharkey,et al.  The "Test and Select" Approach to Ensemble Combination , 2000, Multiple Classifier Systems.

[2]  Luis Angel,et al.  Factorial design for robustness evaluation of fractional PID controllers , 2015, IEEE Latin America Transactions.

[3]  Anjali Gupta,et al.  Optimization of stacking ensemble Configuration based on various metahueristic algorithms , 2014, 2014 IEEE International Advance Computing Conference (IACC).

[4]  Ruimin Xu,et al.  A Large-Signal Statistical Model and Yield Estimation of GaN HEMTs Based on Response Surface Methodology , 2016, IEEE Microwave and Wireless Components Letters.

[5]  Xizhao Wang,et al.  Discrete differential evolutions for the discounted {0-1} knapsack problem , 2017, Int. J. Bio Inspired Comput..

[6]  Ashfaqur Rahman,et al.  Ensemble classifier generation using non-uniform layered clustering and Genetic Algorithm , 2013, Knowl. Based Syst..

[7]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[8]  Robert Sabourin,et al.  Pareto analysis for the selection of classifier ensembles , 2008, GECCO '08.

[9]  Jingbo He,et al.  A dynamic selection ensemble method for target recognition based on clustering and randomized reference classifier , 2019, Int. J. Mach. Learn. Cybern..

[10]  Nor Asiah Muhamad,et al.  Application of response surface methodology for optimizing the oxidative stability of natural ester oil using mixed antioxidants , 2017, IEEE Transactions on Dielectrics and Electrical Insulation.

[11]  Robert Sabourin,et al.  Overfitting cautious selection of classifier ensembles with genetic algorithms , 2009, Inf. Fusion.

[12]  Douglas C. Montgomery,et al.  Response Surface Methodology: Process and Product Optimization Using Designed Experiments , 1995 .

[13]  Kaisa Miettinen,et al.  Nonlinear multiobjective optimization , 1998, International series in operations research and management science.

[14]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[15]  Myoung-Jong Kim,et al.  Classifiers selection in ensembles using genetic algorithms for bankruptcy prediction , 2012, Expert Syst. Appl..

[16]  Jiao Li-cheng,et al.  Intelligent particle swarm optimization in multiobjective optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[17]  Chongzhao Han,et al.  Experiments with Feature-Prior Hybrid Ensemble Method for Classification , 2014, 2014 Tenth International Conference on Computational Intelligence and Security.

[18]  M Reyes Sierra,et al.  Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art , 2006 .

[19]  Lei Xi,et al.  A novel ensemble algorithm for biomedical classification based on Ant Colony Optimization , 2011, Appl. Soft Comput..

[20]  Dominique M. M.-P Schreurs,et al.  Compact Behavioral Models of Nonlinear Active Devices Using Response Surface Methodology , 2015, IEEE Transactions on Microwave Theory and Techniques.

[21]  Vasudha Bhatnagar,et al.  Towards an optimally pruned classifier ensemble , 2014, International Journal of Machine Learning and Cybernetics.

[22]  Mahdi Eftekhari,et al.  A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches , 2015, Appl. Soft Comput..

[23]  Sushanta Paul,et al.  Importance of Voltage Reduction and Optimal Voltage Setting During Reactive Power Compensation , 2014, IEEE Transactions on Power Delivery.

[24]  Joaquim F. Martins-Filho,et al.  New Graph Model to Design Optical Networks , 2015, IEEE Communications Letters.

[25]  S. H. Zahiri,et al.  Application of inclined planes system optimization on data clustering , 2013, 2013 First Iranian Conference on Pattern Recognition and Image Analysis (PRIA).

[26]  Chee Peng Lim,et al.  A multi-objective evolutionary algorithm-based ensemble optimizer for feature selection and classification with neural network models , 2014, Neurocomputing.

[27]  Ganapati Panda,et al.  A survey on nature inspired metaheuristic algorithms for partitional clustering , 2014, Swarm Evol. Comput..

[28]  Rajeev Srivastava,et al.  Filter vs. Wrapper approach for optimum gene selection of high dimensional gene expression dataset: An analysis with cancer datasets , 2014, 2014 International Conference on High Performance Computing and Applications (ICHPCA).

[29]  Carlos A. Coello Coello,et al.  Handling multiple objectives with particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[30]  Yu-Lin He,et al.  Exact and approximate algorithms for discounted {0-1} knapsack problem , 2016, Inf. Sci..

[31]  Derek A. Linkens,et al.  Adaptive Weighted Particle Swarm Optimisation for Multi-objective Optimal Design of Alloy Steels , 2004, PPSN.

[32]  G. Yule.  On the Association of Attributes in Statistics: With Illustrations from the Material of the Childhood Society, &c , 1900 .

[33]  Prashant Singh Rana,et al.  N-semble: neural network based ensemble approach , 2019, Int. J. Mach. Learn. Cybern..

[34]  Gerd Gigerenzer,et al.  Heuristic decision making. , 2011, Annual review of psychology.

[35]  S. Kanmani,et al.  Optimization of stacking ensemble configurations through Artificial Bee Colony algorithm , 2013, Swarm Evol. Comput..

[36]  Hamideh Afsarmanesh,et al.  Boosting for multiclass semi-supervised learning , 2014, Pattern Recognit. Lett..

[37]  Hadi Shahraki,et al.  Fuzzy decision function estimation using fuzzified particle swarm optimization , 2017, Int. J. Mach. Learn. Cybern..

[38]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[39]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[40]  Yen-Sheng Chen,et al.  Application of multi-objective fractional factorial design for ultra-wideband antennas with uniform gain and high fidelity , 2015 .

[41]  Witold Pedrycz,et al.  A Study on Relationship Between Generalization Abilities and Fuzziness of Base Classifiers in Ensemble Learning , 2015, IEEE Transactions on Fuzzy Systems.