A Novel Framework to Design Fuzzy Rule-Based Ensembles Using Diversity Induction and Evolutionary Algorithms-Based Classifier Selection and Fusion

Fuzzy rule-based systems have shown a high capability of knowledge extraction and representation when modeling complex, non-linear classification problems. However, they suffer from the so-called curse of dimensionality when applied to high dimensional datasets, which consist of a large number of variables and/or examples. Multiclassification systems have shown to be a good approach to deal with this kind of problems. In this contribution, we propose an multiclassification system-based global framework allowing fuzzy rule-based systems to deal with high dimensional datasets avoiding the curse of dimensionality. Having this goal in mind, the proposed framework will incorporate several multiclassification system methodologies as well as evolutionary algorithms to design fuzzy rule-based multiclassification systems. The proposed framework follows a two-stage structure: 1) fuzzy rule-based multiclassification system design from classical and advanced multiclassification system design approaches, and 2) novel designs of evolutionary component classifier combination. By using our methodology, different fuzzy rule-based multiclassification systems can be designed dealing with several aspects such as improvement of the performance in terms of accuracy, and obtaining a good accuracy-complexity trade-off.

[1]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[2]  Jerzy Stefanowski,et al.  An Experimental Study of Methods Combining Multiple Classifiers-Diversified both by Feature Selection and Bootstrap Sampling , 2005 .

[3]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[4]  Hiroyuki Honda,et al.  Lymphoma Prognostication from Expression Profiling Using a Combination Method of Boosting and Projective Adaptive Resonance Theory , 2006 .

[5]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[6]  Daniel Hernández-Lobato,et al.  How large should ensembles of classifiers be? , 2013, Pattern Recognit..

[7]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[8]  Francisco Herrera,et al.  Other Genetic Fuzzy Rule-Based System Paradigms , 2001 .

[9]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[10]  James C. Bezdek,et al.  Decision templates for multiple classifier fusion: an experimental comparison , 2001, Pattern Recognit..

[11]  Lawrence O. Hall,et al.  Ensembles of Fuzzy Classifiers , 2007, 2007 IEEE International Fuzzy Systems Conference.

[12]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[13]  Francisco Herrera,et al.  Genetic fuzzy systems: taxonomy, current research trends and prospects , 2008, Evol. Intell..

[14]  Jon Atli Benediktsson,et al.  Multiple Classifier Systems , 2015, Lecture Notes in Computer Science.

[15]  Oscar Cordón,et al.  Comparing two genetic overproduce-and-choose strategies for fuzzy rule-based multiclassification systems generated by bagging and mutual information-based feature selection , 2010, Int. J. Hybrid Intell. Syst..

[16]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[17]  Kevin W. Bowyer,et al.  Combination of Multiple Classifiers Using Local Accuracy Estimates , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Robert Sabourin,et al.  A dynamic overproduce-and-choose strategy for the selection of classifier ensembles , 2008, Pattern Recognit..

[19]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[20]  Francisco Herrera,et al.  A hierarchical knowledge-based environment for linguistic modeling: models and iterative methodology , 2003, Fuzzy Sets Syst..

[21]  María José del Jesús,et al.  Niching genetic feature selection algorithms applied to the design of fuzzy rule-based classification systems , 2007, 2007 IEEE International Fuzzy Systems Conference.

[22]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[23]  Mykola Pechenizkiy,et al.  Diversity in search strategies for ensemble feature selection , 2005, Inf. Fusion.

[24]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[25]  Luis Magdalena,et al.  Interpretability Improvements to Find the Balance Interpretability-Accuracy in Fuzzy Modeling: An Overview , 2003 .

[26]  Jesús Alcalá-Fdez,et al.  KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework , 2011, J. Multiple Valued Log. Soft Comput..

[27]  F. Herrera,et al.  A proposal on reasoning methods in fuzzy rule-based classification systems , 1999 .

[28]  Ronald R. Yager,et al.  On the construction of hierarchical fuzzy systems models , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[29]  Mauricio G. C. Resende,et al.  Greedy Randomized Adaptive Search Procedures , 1995, J. Glob. Optim..

[30]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[31]  Hisao Ishibuchi,et al.  Classification and modeling with linguistic information granules - advanced approaches to linguistic data mining , 2004, Advanced information processing.

[32]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[33]  Hisao Ishibuchi,et al.  Genetic rule selection with a multi-classifier coding scheme for ensemble classifier design , 2007, Int. J. Hybrid Intell. Syst..

[34]  Thomas G. Dietterich,et al.  Pruning Adaptive Boosting , 1997, ICML.

[35]  José M. Alonso,et al.  Looking for a good fuzzy system interpretability index: An experimental approach , 2009, Int. J. Approx. Reason..

[36]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[37]  Ludmila I. Kuncheva,et al.  "Fuzzy" versus "nonfuzzy" in combining classifiers designed by Boosting , 2003, IEEE Trans. Fuzzy Syst..

[38]  Witold Pedrycz,et al.  Boosting of granular models , 2006, Fuzzy Sets Syst..

[39]  Hisao Ishibuchi,et al.  Designing Fuzzy Ensemble Classifiers by Evolutionary Multiobjective Optimization with an Entropy-Based Diversity Criterion , 2006, 2006 Sixth International Conference on Hybrid Intelligent Systems (HIS'06).

[40]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[41]  Cezary Z. Janikow,et al.  Fuzzy decision trees: issues and methods , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[42]  Oscar Cordón,et al.  Multiobjective genetic classifier selection for random oracles fuzzy rule-based classifier ensembles: How beneficial is the additional diversity? , 2013, Knowl. Based Syst..

[43]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[44]  Francisco Herrera,et al.  Genetic Fuzzy Systems - Evolutionary Tuning and Learning of Fuzzy Knowledge Bases , 2002, Advances in Fuzzy Systems - Applications and Theory.

[45]  Saso Dzeroski,et al.  Combining Bagging and Random Subspaces to Create Better Ensembles , 2007, IDA.

[46]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[47]  Hisao Ishibuchi,et al.  Voting in fuzzy rule-based systems for pattern classification problems , 1999, Fuzzy Sets Syst..

[48]  Thomas G. Dietterich.  An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization , 2000, Machine Learning.

[49]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[50]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[51]  José M. Alonso,et al.  A multiclassifier approach for topology-based WiFi indoor localization , 2013, Soft Computing.

[52]  Eyke Hüllermeier,et al.  FURIA: an algorithm for unordered fuzzy rule induction , 2009, Data Mining and Knowledge Discovery.

[53]  Oscar Cordón,et al.  On Designing Fuzzy Rule-Based Multiclassification Systems by Combining Furia with Bagging and Feature Selection , 2011, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[54]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[55]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[56]  O. Cordón,et al.  A Genetic Fuzzy Linguistic Combination Method for Fuzzy Rule-Based Multiclassifiers , 2013, IEEE Transactions on Fuzzy Systems.

[57]  John Shawe-Taylor,et al.  Advances in Intelligent Data Analysis VII, 7th International Symposium on Intelligent Data Analysis, IDA 2007, Ljubljana, Slovenia, September 6-8, 2007, Proceedings , 2007, IDA.

[58]  Juan José Rodríguez Diez,et al.  Naïve Bayes Ensembles with a Random Oracle , 2007, MCS.

[59]  Ronald R. Yager,et al.  Essentials of fuzzy modeling and control , 1994 .

[60]  Hisao Ishibuchi,et al.  Evolutionary multiobjective optimization for the design of fuzzy rule-based ensemble classifiers , 2006, Int. J. Hybrid Intell. Syst..

[61]  B.V. Dasarathy,et al.  A composite classifier system design: Concepts and methodology , 1979, Proceedings of the IEEE.

[62]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[63]  Tin Kam Ho,et al.  The Random Subspace Method for Constructing Decision Forests , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[65]  Oscar Cordón,et al.  International Journal of Approximate Reasoning a Historical Review of Evolutionary Learning Methods for Mamdani-type Fuzzy Rule-based Systems: Designing Interpretable Genetic Fuzzy Systems , 2022 .

[66]  Juan José Rodríguez Diez,et al.  Classifier Ensembles with a Random Linear Oracle , 2007, IEEE Transactions on Knowledge and Data Engineering.

[67]  Piero P. Bonissone,et al.  A fuzzy random forest , 2010, Int. J. Approx. Reason..

[68]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[69]  Noel E. Sharkey,et al.  The "Test and Select" Approach to Ensemble Combination , 2000, Multiple Classifier Systems.

[70]  William B. Yates,et al.  Engineering Multiversion Neural-Net Systems , 1996, Neural Computation.

[71]  Christophe Marsala,et al.  Data mining with ensembles of fuzzy decision trees , 2009, 2009 IEEE Symposium on Computational Intelligence and Data Mining.

[72]  J. Casillas.  Interpretability issues in fuzzy modeling , 2003 .

[73]  Oscar Cordón,et al.  A Study on the Use of Multiobjective Genetic Algorithms for Classifier Selection in FURIA-based Fuzzy Multiclassifiers , 2012, Int. J. Comput. Intell. Syst..

[74]  Vicenç Torra,et al.  A review of the construction of hierarchical fuzzy systems , 2002, Int. J. Intell. Syst..

[75]  Paul M. Frank,et al.  Hierarchical fuzzy control of multivariable systems , 1995 .

[76]  Francisco Herrera,et al.  Ten years of genetic fuzzy systems: current framework and new trends , 2004, Fuzzy Sets Syst..

[77]  Hakan Altinçay,et al.  Classifier subset selection for biomedical named entity recognition , 2009, Applied Intelligence.

[78]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation) , 2006 .