Fuzzy Ensemble Design through Multi-Objective Fuzzy Rule Selection

Themain advantage of evolutionary multi-objective optimization (EMO) over classical approaches is that a variety of non-dominated solutions with a wide range of objective values can be simultaneously obtained by a single run of an EMO algorithm. In this chapter, we show how this advantage can be utilized in the design of fuzzy ensemble classifiers. First we explain three objectives in multi-objective formulations of fuzzy rule selection. One is accuracy maximization and the others are complexity minimization. Next we demonstrate that a number of non-dominated rule sets (i.e., fuzzy classifiers) are obtained along the accuracy-complexity tradeoff surface from multi-objective fuzzy rule selection problems. Then we examine the effect of combining multiple non-dominated fuzzy classifiers into a single ensemble classifier. Experimental results clearly show that the combination into ensemble classifiers improves the classification ability of individual fuzzy classifiers for some data sets.

[1]  Bernhard Sendhoff,et al.  Neural network regularization and ensembling using multi-objective evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[2]  Hideo Tanaka,et al.  Construction of fuzzy classification systems with rectangular fuzzy rules using genetic algorithms , 1994, CVPR 1994.

[3]  F. Gomide,et al.  Ten years of genetic fuzzy systems: current framework and new trends , 2001, Proceedings Joint 9th IFSA World Congress and 20th NAFIPS International Conference (Cat. No. 01TH8569).

[4]  Roel Wieringa,et al.  An integrated framework for ought-to-be and ought-to-do constraints , 2004, Artificial Intelligence and Law.

[5]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[6]  Antonio F. Gómez-Skarmeta,et al.  Accurate, Transparent, and Compact Fuzzy Models for Function Approximation and Dynamic Modeling through Multi-objective Evolutionary Optimization , 2001, EMO.

[7]  J. Casillas.  Interpretability issues in fuzzy modeling , 2003 .

[8]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[9]  Kim-Fung Man,et al.  Agent-based evolutionary approach for interpretable rule-based knowledge extraction , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[10]  Hisao Ishibuchi,et al.  Evolutionary Multiobjective Knowledge Extraction for High-Dimensional Pattern Classification Problems , 2004, PPSN.

[11]  Bernhard Sendhoff,et al.  EVOLUTIONARY MULTI-OBJECTIVE OPTIMIZATION APPROACH TO CONSTRUCTING NEURAL NETWORK ENSEMBLES FOR REGRESSION , 2004 .

[12]  Tapio Elomaa,et al.  General and Efficient Multisplitting of Numerical Attributes , 1999, Machine Learning.

[13]  Hisao Ishibuchi,et al.  Rule weight specification in fuzzy rule-based classification systems , 2005, IEEE Transactions on Fuzzy Systems.

[14]  Hisao Ishibuchi,et al.  Three-objective genetics-based machine learning for linguistic rule extraction , 2001, Inf. Sci..

[15]  Hisao Ishibuchi,et al.  Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining , 2004, Fuzzy Sets Syst..

[16]  Luis Magdalena,et al.  A Multiobjective Genetic Learning Process for joint Feature Selection and Granularity and Contexts Learning in Fuzzy Rule-Based Classification Systems , 2003 .

[17]  Antonio González Muñoz,et al.  SLAVE: a genetic learning system based on an iterative approach , 1999, IEEE Trans. Fuzzy Syst..

[18]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[19]  Gary B. Lamont,et al.  Applications Of Multi-Objective Evolutionary Algorithms , 2004 .

[20]  Hisao Ishibuchi,et al.  Classification and modeling with linguistic information granules - advanced approaches to linguistic data mining , 2004, Advanced information processing.

[21]  Luiz Eduardo Soares de Oliveira,et al.  Multi-objective Genetic Algorithms to Create Ensemble of Classifiers , 2005, EMO.

[22]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[23]  Hisao Ishibuchi,et al.  Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems , 1997, Fuzzy Sets Syst..

[24]  Hisao Ishibuchi,et al.  Voting in fuzzy rule-based systems for pattern classification problems , 1999, Fuzzy Sets Syst..

[25]  Alberto Maria Segre,et al.  Programs for Machine Learning , 1994 .

[26]  Hisao Ishibuchi,et al.  Selecting fuzzy if-then rules for classification problems using genetic algorithms , 1995, IEEE Trans. Fuzzy Syst..

[27]  Hisao Ishibuchi,et al.  Hybridization of fuzzy GBML approaches for pattern classification problems , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[29]  Hisao Ishibuchi,et al.  Comparison of Heuristic Criteria for Fuzzy Rule Selection in Classification Problems , 2004, Fuzzy Optim. Decis. Mak..

[30]  Hisao Ishibuchi,et al.  Effects of Three-Objective Genetic Rule Selection on the Generalization Ability of Fuzzy Rule-Based Systems , 2003, EMO.

[31]  Kim-Fung Man,et al.  Multi-objective hierarchical genetic algorithm for interpretable fuzzy rule-based knowledge extraction , 2005, Fuzzy Sets Syst..

[32]  Antonio F. Gómez-Skarmeta,et al.  Accurate, transparent and compact fuzzy models by multi-objective evolutionary algorithms , 2003 .

[33]  Luiz Eduardo Soares de Oliveira,et al.  Feature selection for ensembles:a hierarchical multi-objective genetic algorithm approach , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[34]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[35]  Xin Yao,et al.  Evolutionary framework for the construction of diverse hybrid ensembles , 2005, ESANN.

[36]  J. Ross Quinlan,et al.  Improved Use of Continuous Attributes in C4.5 , 1996, J. Artif. Intell. Res..

[37]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[38]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[39]  Hisao Ishibuchi,et al.  Evolutionary Multiobjective Optimization for Generating an Ensemble of Fuzzy Rule-Based Classifiers , 2003, GECCO.