Ensemble pruning based on oblivious Chained Tabu Searches

Ensemble pruning is an effective phase for ensemble methods to increase the predictive performance and to decrease computational overheads. This paper represents a novel ensemble pruning method named EPCTS(Ensemble Pruning via Chained Tabu Searches). EPCTS applies a chain of tabu searches for choosing models of ensemble progressively, until the best subset of them is found. These tabu searches are customized with the proposed strategy dubbed as "Periodic Oblivion". This strategy revokes interdict of all tabu answers in the defined periods. EPCTS is compared with analogous ensemble pruning methods for pruning a balanced heterogeneous ensemble, focusing on 20 problems. Experimental results demonstrate that EPCTS leads to 2.65% averaged improvement in the accuracy of pruned ensemble, compared to others. Further, EPCTS leads to reduce computational overheads with dropping redundant and useless models from the ensemble. Moreover, one of the crucial issues in the ensemble learning field is making the decision to choose the type of base classifiers constructing desired ensemble. Considering the importance of the issue and due to the effectiveness of EPCTS in about 75% of datasets, EPCTS is suggested as a general tool for recognizing the type of base classifiers.

[1]  Khaled S. Al-Sultan,et al.  A tabu search-based algorithm for the fuzzy clustering problem , 1997, Pattern Recognit..

[2]  Chen Lin,et al.  LibD3C: Ensemble classifiers with a clustering and dynamic selection strategy , 2014, Neurocomputing.

[3]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[4]  Vipin Kumar,et al.  Introduction to Data Mining , 2022, Data Mining and Machine Learning Applications.

[5]  Grigorios Tsoumakas,et al.  An ensemble uncertainty aware measure for directed hill climbing ensemble pruning , 2010, Machine Learning.

[6]  Wei Tang,et al.  Selective Ensemble of Decision Trees , 2003, RSFDGrC.

[7]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[8]  Dag Haugland,et al.  A tabu search heuristic for the vehicle routing problem with time windows and split deliveries , 2004, Comput. Oper. Res..

[9]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[10]  Kalyanmoy Deb,et al.  Multi-objective optimization using evolutionary algorithms , 2001, Wiley-Interscience series in systems and optimization.

[11]  Fabio Roli,et al.  Design of effective multiple classifier systems by clustering of classifiers , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[12]  Asif Ekbal,et al.  A multiobjective simulated annealing approach for classifier ensemble: Named entity recognition in Indian languages as case studies , 2011, Expert Syst. Appl..

[13]  Asif Ekbal,et al.  Simulated annealing based classifier ensemble techniques: Application to part of speech tagging , 2013, Inf. Fusion.

[14]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[15]  Teodor Gabriel Crainic,et al.  A metaheuristic for stochastic service network design , 2010, J. Heuristics.

[16]  Foster J. Provost,et al.  A Survey of Methods for Scaling Up Inductive Algorithms , 1999, Data Mining and Knowledge Discovery.

[17]  R. Anitha,et al.  Malware detection by pruning of parallel ensembles using harmony search , 2013, Pattern Recognit. Lett..

[18]  Grzegorz Waligóra,et al.  Tabu search for multi-mode resource-constrained project scheduling with schedule-dependent setup times , 2008, Eur. J. Oper. Res..

[19]  Philip K. Chan,et al.  Meta-learning in distributed data mining systems: Issues and approaches , 2007 .

[20]  Zhi-Hua Zhou,et al.  Ensemble Methods: Foundations and Algorithms , 2012 .

[21]  Dong Cao,et al.  Coordinating production planning in cellular manufacturing environment using Tabu search , 2004, Comput. Ind. Eng..

[22]  Fred W. Glover,et al.  Future paths for integer programming and links to artificial intelligence , 1986, Comput. Oper. Res..

[23]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[24]  Asif Ekbal,et al.  Weighted Vote-Based Classifier Ensemble for Named Entity Recognition: A Genetic Algorithm-Based Approach , 2011, TALIP.

[25]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[26]  Rich Caruana,et al.  Ensemble selection from libraries of models , 2004, ICML.

[27]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[28]  Hedieh Sajedi,et al.  Human-inspired ensemble pruning using hill climbing algorithm , 2013, 2013 3rd Joint Conference of AI & Robotics and 5th RoboCup Iran Open International Symposium.

[29]  Minghe Sun.  Solving the uncapacitated facility location problem using tabu search , 2006, Comput. Oper. Res..

[30]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[31]  Ujjwal Maulik,et al.  A Simulated Annealing-Based Multiobjective Optimization Algorithm: AMOSA , 2008, IEEE Transactions on Evolutionary Computation.

[32]  Grigorios Tsoumakas,et al.  Pruning an ensemble of classifiers via reinforcement learning , 2009, Neurocomputing.

[33]  Ming Fan,et al.  Instance-based ensemble pruning for imbalanced learning , 2015, Intell. Data Anal..

[34]  Lawrence O. Hall,et al.  Ensemble diversity measures and their application to thinning , 2004, Inf. Fusion.

[35]  José Brandão,et al.  A deterministic tabu search algorithm for the fleet size and mix vehicle routing problem , 2009, Eur. J. Oper. Res..

[36]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[37]  Asif Ekbal,et al.  Combining multiple classifiers using vote based classifier ensemble technique for named entity recognition , 2013, Data Knowl. Eng..

[38]  Jin-Kao Hao,et al.  Adaptive Tabu Search for course timetabling , 2010, Eur. J. Oper. Res..

[39]  Chris D. Nugent,et al.  Pruning extensions to stacking , 2006, Intell. Data Anal..

[40]  Çagdas Hakan Aladag,et al.  The effect of neighborhood structures on tabu search algorithm in solving course timetabling problem , 2009, Expert Syst. Appl..

[41]  Zülal Güngör,et al.  K-Harmonic means data clustering with tabu-search method , 2008 .

[42]  Fred W. Glover,et al.  A cooperative parallel tabu search algorithm for the quadratic assignment problem , 2009, Eur. J. Oper. Res..