Evolutionary approaches to fuzzy modelling for classification

An overview of the application of evolutionary computation to fuzzy knowledge discovery is presented. This is set in one of two contexts: overcoming the knowledge acquisition bottleneck in the development of intelligent reasoning systems, and in the data mining of databases where the aim is the discovery of new knowledge. The different strategies utilizing evolutionary algorithms for knowledge acquisition are abstracted from the work reviewed. The simplest strategy runs an evolutionary algorithm once, while the iterative rule learning approach runs several evolutionary algorithms in succession, with the output from each considered a partial solution. Ensembles are formed by combining several classifiers generated by evolutionary techniques, while co-evolution is often used for evolving rule bases and associated membership functions simultaneously. The associated strengths and limitations of these induction strategies are compared and discussed. Ways in which evolutionary techniques have been adapted to satisfy the common evaluation criteria of the induced knowledge—classification accuracy, comprehensibility and novelty value—are also considered. The review concludes by highlighting common limitations of the experimental methodology used and indicating ways of resolving them.

[1]  Russell C. Eberhart,et al.  Implementation of evolutionary fuzzy systems , 1999, IEEE Trans. Fuzzy Syst..

[2]  Xavier Llorà,et al.  Evolution of Decision Trees , 2001 .

[3]  L. Darrell Whitley,et al.  Genetic Approach to Feature Selection for Ensemble Creation , 1999, GECCO.

[4]  Se-Young Oh,et al.  Automatic rule generation for fuzzy logic controllers using rule-level co-evolution of subpopulations , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[5]  Vili Podgorelec,et al.  Decision tree's induction strategies evaluated on a hard real world problem , 2000, Proceedings 13th IEEE Symposium on Computer-Based Medical Systems. CBMS 2000.

[6]  Paul W. Munro,et al.  Improving Committee Diagnosis with Resampling Techniques , 1995, NIPS.

[7]  Frank Hoffmann,et al.  Evolutionary algorithms for fuzzy control system design , 2001, Proc. IEEE.

[8]  Hisao Ishibuchi,et al.  Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining , 2004, Fuzzy Sets Syst..

[9]  Larry Bull,et al.  A Genetic Programming-based Classifier System , 1999, GECCO.

[10]  Antonio F. Gómez-Skarmeta,et al.  Approximative fuzzy rules approaches for classification with hybrid-GA techniques , 2001, Inf. Sci..

[11]  David E. Goldberg,et al.  Accuracy, Parsimony, and Generality in Evolutionary Learning Systems via Multiobjective Selection , 2002, IWLCS.

[12]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[13]  Kenneth A. De Jong,et al.  Using genetic algorithms for concept learning , 1993, Machine Learning.

[14]  Thomas Bäck,et al.  Evolutionary Algorithms in Theory and Practice , 1996 .

[15]  Moshe Sipper,et al.  Fuzzy CoCo: a cooperative-coevolutionary approach to fuzzy modeling , 2001, IEEE Trans. Fuzzy Syst..

[16]  E. H. Mamdani,et al.  Advances in the linguistic synthesis of fuzzy controllers , 1976 .

[17]  Kwong-Sak Leung,et al.  Inducing Logic Programs With Genetic Algorithms: The Genetic Logic Programming System , 1995, IEEE Expert.

[18]  Ester Bernadó-Mansilla,et al.  Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks , 2003, Evolutionary Computation.

[19]  Hisao Ishibuchi,et al.  Evolutionary algorithms for constructing linguistic rule-based systems for high-dimensional pattern classification problems , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[20]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[21]  Jin-Fu Chang,et al.  Knowledge Representation Using Fuzzy Petri Nets , 1990, IEEE Trans. Knowl. Data Eng..

[22]  Moshe Sipper,et al.  A fuzzy-genetic approach to breast cancer diagnosis , 1999, Artif. Intell. Medicine.

[23]  L. Darrell Whitley,et al.  An overview of evolutionary algorithms: practical issues and common pitfalls , 2001, Inf. Softw. Technol..

[24]  Hisao Ishibuchi,et al.  A hybrid fuzzy GBML algorithm for designing compact fuzzy rule-based classification systems , 2000, Ninth IEEE International Conference on Fuzzy Systems. FUZZ- IEEE 2000 (Cat. No.00CH37063).

[25]  Alex A. Freitas,et al.  A distributed-population genetic algorithm for discovering interesting prediction rules , 2002 .

[26]  Marco Colombetti,et al.  What Is a Learning Classifier System? , 1999, Learning Classifier Systems.

[27]  Wynne Hsu,et al.  Using General Impressions to Analyze Discovered Classification Rules , 1997, KDD.

[28]  Antonio González Muñoz,et al.  Multi-stage genetic fuzzy systems based on the iterative rule learning approach , 1997 .

[29]  J. Pollack,et al.  Coevolutionary dynamics in a minimal substrate , 2001 .

[30]  Thomas G. Dietterich Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[31]  Antonio González Muñoz,et al.  SLAVE: a genetic learning system based on an iterative approach , 1999, IEEE Trans. Fuzzy Syst..

[32]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[33]  Vladan Babovic,et al.  Genetic Programming, Ensemble Methods and the Bias/Variance Tradeoff - Introductory Investigations , 2000, EuroGP.

[34]  A. Krone,et al.  A hybrid evolutionary search concept for data-based generation of relevant fuzzy rules in high dimensional spaces , 1999, FUZZ-IEEE'99. 1999 IEEE International Fuzzy Systems. Conference Proceedings (Cat. No.99CH36315).

[35]  Isak Gath,et al.  Unsupervised Optimal Fuzzy Clustering , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[37]  Wolfgang Stolzmann,et al.  Anticipatory Classifier Systems: An introduction , 2001 .

[38]  Marek Kretowski,et al.  Discovery of Decision Rules from Databases: An Evolutionary Approach , 1998, PKDD.

[39]  Qiang Shen,et al.  From approximative to descriptive fuzzy classifiers , 2002, IEEE Trans. Fuzzy Syst..

[40]  Rafee Ebrahim,et al.  Fuzzy logic programming , 2001, Fuzzy Sets Syst..

[41]  Miguel Toro,et al.  Discovering hierarchical decision rules with evolutive algorithms in supervised learning , 2000, Int. J. Comput. Syst. Signals.

[42]  Simon Kasif,et al.  A System for Induction of Oblique Decision Trees , 1994, J. Artif. Intell. Res..

[43]  A. Engelbrecht,et al.  Searching the forest: using decision trees as building blocks for evolutionary search in classification databases , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[44]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[45]  J. Ross Quinlan,et al.  Learning logical definitions from relations , 1990, Machine Learning.

[46]  Alastair Smith,et al.  How not to do it , 2005 .

[47]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[48]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[49]  H. Ishibuchi,et al.  A fuzzy classifier system that generates fuzzy if-then rules for pattern classification problems , 1995, Proceedings of 1995 IEEE International Conference on Evolutionary Computation.

[50]  William B. Langdon,et al.  Genetic programming for combining classifiers , 2001 .

[51]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[52]  Einoshin Suzuki,et al.  Discovery of Surprising Exception Rules Based on Intensity of Implication , 1998, PKDD.

[53]  James M. Keller,et al.  A possibilistic approach to clustering , 1993, IEEE Trans. Fuzzy Syst..

[54]  Ulrich Bodenhofer,et al.  FS-FOIL: an inductive learning method for extracting interpretable fuzzy descriptions , 2003, Int. J. Approx. Reason..

[55]  Francisco Herrera,et al.  Encouraging Cooperation in the Genetic Iterative Rule Learning Approach for Qualitative Modeling , 1999 .

[56]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[58]  Federico Divina,et al.  Evolutionary Concept Learning , 2002, GECCO.

[59]  Cezary Z. Janikow,et al.  A genetic algorithm method for optimizing fuzzy decision trees , 1996 .

[60]  Hisao Ishibuchi,et al.  Performance evaluation of fuzzy classifier systems for multidimensional pattern classification problems , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[61]  Francisco Herrera,et al.  Cooperative Coevolution for Learning Fuzzy Rule-Based Systems , 2001, Artificial Evolution.

[62]  Kwong-Sak Leung,et al.  FF99: a novel fuzzy first-order logic learning system , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[63]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[64]  Trevor P Martin,et al.  Fril- Fuzzy and Evidential Reasoning in Artificial Intelligence , 1995 .

[65]  Ching-Chi Hsu,et al.  GEC: An Evolutionary Approach for Evolving Classifiers , 2002, PAKDD.

[66]  Andrea Bonarini,et al.  An Introduction to Learning Fuzzy Classifier Systems , 1999, Learning Classifier Systems.

[67]  Jem J. Rowland,et al.  Generalisation and Model Selection in Supervised Learning with Evolutionary Computation , 2003, EvoWorkshops.

[68]  C. K. Mohan,et al.  ClaDia: a fuzzy classifier system for disease diagnosis , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[69]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[70]  Bruce G. Buchanan,et al.  Principles of Rule-Based Expert Systems , 1982, Adv. Comput..

[71]  Francisco Herrera,et al.  Genetic fuzzy systems. New developments , 2004, Fuzzy Sets Syst..

[72]  Helmut A. Mayer,et al.  Extraction of compact rule sets from evolutionary designed artificial neural networks , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[73]  Alex A. Freitas,et al.  A Genetic Algorithm for Generalized Rule Induction , 1999 .

[74]  Tomoharu Nakashima,et al.  A fuzzy genetics-based machine learning method for designing linguistic classification systems with high comprehensibility , 1999, ICONIP'99. ANZIIS'99 & ANNES'99 & ACNN'99. 6th International Conference on Neural Information Processing. Proceedings (Cat. No.99EX378).

[75]  Marco Dorigo,et al.  Swarm intelligence: from natural to artificial systems , 1999 .

[76]  Rudy Setiono,et al.  Generating concise and accurate classification rules for breast cancer diagnosis , 2000, Artif. Intell. Medicine.

[77]  Gary B. Lamont,et al.  Multiobjective Evolutionary Algorithms: Analyzing the State-of-the-Art , 2000, Evolutionary Computation.

[78]  Alex Alves Freitas,et al.  A Genetic Algorithm For Discovering Interesting Fuzzy Prediction Rules: Applications To Science And Technology Data , 2002, GECCO.

[79]  Charles X. Ling,et al.  AUC: A Better Measure than Accuracy in Comparing Learning Algorithms , 2003, Canadian Conference on AI.

[80]  Pier Luca Lanzi,et al.  A Roadmap to the Last Decade of Learning Classifier System Research , 1999, Learning Classifier Systems.

[81]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[82]  I. Hatono,et al.  Fuzzy decision trees by fuzzy ID3 algorithm and its application to diagnosis systems , 1994, Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference.

[83]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[84]  Michael C. Fairhurst,et al.  An evolutionary algorithm for classifier and combination rule selection in multiple classifier systems , 2002, Object recognition supported by user interaction for service robots.

[85]  Man Leung Wong,et al.  A flexible knowledge discovery system using genetic programming and logic grammars , 2001, Decis. Support Syst..

[86]  Marco Dorigo,et al.  Alecsys and the AutonoMouse: Learning to control a real robot by distributed classifier systems , 2004, Machine Learning.

[87]  Tohgoroh Matsui,et al.  An Induction Algorithm Based on Fuzzy Logic Programming , 1999, PAKDD.

[88]  John H. Holmes,et al.  Learning Classifier Systems Applied to Knowledge Discovery in Clinical Research Databases , 1999, Learning Classifier Systems.

[89]  Kumar Chellapilla,et al.  Data mining using genetic programming: the implications of parsimony on generalization error , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[90]  Gilles Venturini,et al.  Learning First Order Logic Rules with a Genetic Algorithm , 1995, KDD.

[91]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[92]  Howard J. Hamilton,et al.  Heuristic for Ranking the Interestigness of Discovered Knowledge , 1999, PAKDD.

[93]  J. Juan Liu,et al.  An extended genetic rule induction algorithm , 2000, CEC.

[94]  Frank Hoffmann,et al.  Combining boosting and evolutionary algorithms for learning of fuzzy classification rules , 2004, Fuzzy Sets Syst..

[95]  Tim Kovacs,et al.  Two Views of Classifier Systems , 2001, IWLCS.

[96]  Larry Bull,et al.  Feature Construction and Selection Using Genetic Programming and a Genetic Algorithm , 2003, EuroGP.

[97]  F. Herrera,et al.  Genetic learning of fuzzy rule‐based classification systems cooperating with fuzzy reasoning methods , 1998 .

[98]  E. F. Codd,et al.  Cellular automata , 1968 .

[99]  Randall Davis,et al.  An overview of production systems , 1975 .

[100]  D. H. Widyantoro,et al.  An entropy-based adaptive genetic algorithm for learning classification rules , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[101]  Filippo Menczer,et al.  Meta-evolutionary ensembles , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[102]  Kaoru Hirota,et al.  Industrial Applications of Fuzzy Technology , 1993, Springer Japan.

[103]  Lucien Duckstein,et al.  Fuzzy Rule-Based Modeling with Applications to Geophysical, Biological and Engineering Systems , 1995 .

[104]  Pat Langley,et al.  Crafting Papers on Machine Learning , 2000, ICML.

[105]  Kwong-Sak Leung,et al.  An induction system that learns programs in different programming languages using genetic programming and logic grammars , 1995, Proceedings of 7th IEEE International Conference on Tools with Artificial Intelligence.

[106]  Jordan B. Pollack,et al.  A Mathematical Framework for the Study of Coevolution , 2002, FOGA.

[107]  Ibrahim Kushchu,et al.  An Evaluation of EvolutionaryGeneralisation in Genetic Programming , 2002, Artificial Intelligence Review.

[108]  Richard K. Belew,et al.  New Methods for Competitive Coevolution , 1997, Evolutionary Computation.

[109]  David B. Fogel,et al.  The Advantages of Evolutionary Computation , 1997, BCEC.

[110]  Yufei Yuan,et al.  A genetic algorithm for generating fuzzy classification rules , 1996, Fuzzy Sets Syst..

[111]  Qiangfu Zhao,et al.  Generation of comprehensible decision trees through evolution of training data , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[112]  Toshinori Munakata Notes on implementing fuzzy sets in Prolog , 1998, Fuzzy Sets Syst..

[113]  M. Lozano,et al.  MOGUL: A methodology to obtain genetic fuzzy rule‐based systems under the iterative rule learning approach , 1999 .

[114]  Donald A. Waterman,et al.  Pattern-Directed Inference Systems , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[115]  Alex A. Freitas,et al.  Discovering interesting prediction rules with a genetic algorithm , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[116]  Antonio F. Gómez-Skarmeta,et al.  Accurate, Transparent, and Compact Fuzzy Models for Function Approximation and Dynamic Modeling through Multi-objective Evolutionary Optimization , 2001, EMO.

[117]  Cezary Z. Janikow,et al.  A knowledge-intensive genetic algorithm for supervised learning , 1993, Machine Learning.

[118]  Alexandre Parodi,et al.  An Efficient Classifier System and Its Experimental Comparison with Two Representative Learning Methods on Three Medical Domains , 1991, ICGA.

[119]  Alex Alves Freitas,et al.  Discovering Fuzzy Classification Rules with Genetic Programming and Co-evolution , 2001, PKDD.

[120]  Jeroen Eggermont,et al.  Evolving Fuzzy Decision Trees with Genetic Programming and Clustering , 2002, EuroGP.

[121]  Deyi Li,et al.  A Fuzzy Prolog Database System , 1990 .

[122]  Nicolas Ragot,et al.  A new hybrid learning method for fuzzy decision trees , 2001, 10th IEEE International Conference on Fuzzy Systems. (Cat. No.01CH37297).

[123]  Stephen Muggleton,et al.  Efficient Induction of Logic Programs , 1990, ALT.

[124]  Hisao Ishibuchi,et al.  Genetic-algorithm-based approaches to the design of fuzzy systems for multi-dimensional pattern classification problems , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[125]  Alex A. Freitas,et al.  A survey of evolutionary algorithms for data mining and knowledge discovery , 2003 .

[126]  Jordan B. Pollack,et al.  Coevolutionary Learning: A Case Study , 1998, ICML.

[127]  John R. Koza,et al.  Concept Formation and Decision Tree Induction Using the Genetic Programming Paradigm , 1990, PPSN.

[128]  Alex Alves Freitas,et al.  On Objective Measures of Rule Surprisingness , 1998, PKDD.

[129]  Chandrika Kamath,et al.  Inducing oblique decision trees with evolutionary algorithms , 2003, IEEE Trans. Evol. Comput..

[130]  Ibrahim Kuscu,et al.  Generalisation and domain specific functions in genetic programming , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[131]  Gilles Venturini,et al.  SIA: A Supervised Inductive Algorithm with Genetic Search for Learning Attributes based Concepts , 1993, ECML.

[132]  Lawrence J. Fogel,et al.  Intelligence Through Simulated Evolution: Forty Years of Evolutionary Programming , 1999 .

[133]  Abraham Silberschatz,et al.  What Makes Patterns Interesting in Knowledge Discovery Systems , 1996, IEEE Trans. Knowl. Data Eng..

[134]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[135]  Jukka Hekanaho DOGMA: A GA-Based Relational Learner , 1998, ILP.

[136]  Jean-Arcady Meyer,et al.  A hierarchical classifier system implementing a motivationally autonomous animat , 1994 .

[137]  Raúl Pérez,et al.  Completeness and consistency conditions for learning fuzzy rules , 1998, Fuzzy Sets Syst..

[138]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[139]  Antonio González,et al.  A learning methodology in uncertain and imprecise environments , 1995 .

[140]  L. Kallel,et al.  Theoretical Aspects of Evolutionary Computing , 2001, Natural Computing Series.

[141]  Stephen F. Smith,et al.  A learning system based on genetic adaptive algorithms , 1980 .

[142]  M. Quafafou,et al.  Learning fuzzy relational descriptions using the logical framework and rough set theory , 1998, 1998 IEEE International Conference on Fuzzy Systems Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36228).