A bi-phased multi-objective genetic algorithm based classifier

Abstract This paper presents a novel Bi-Phased Multi-Objective Genetic Algorithm (BPMOGA) based classification method. It is a Learning Classifier System (LCS) designed for supervised learning tasks. Here we have used Genetic Algorithms (GAs) to discover optimal classifiers from data sets. The objective of the work is to find out a classifier or Complete Rule (CR) which comprises of several Class Specific Rules (CSRs). Phase-I of BPMOGA extracts optimized CSRs in I F − T H E N form by following Michigan approach, without considering interaction among the rules. Phase-II of BPMOGA builds optimized CRs from CSRs by following Pittsburgh way. It combines the advantages of both approaches. Extracted CRs help to build CSRs for the next run of phase-I. Hence, phase-I and phase-II are cyclically related, which is one of the uniqueness of BPMOGA. With the help of twenty one benchmark data sets from the University of California at Irvine (UCI) machine learning repository we have compared performance of BPMOGA based classifier with fourteen GA and non-GA based classifiers. Statistical test shows that the performance of the proposed classifier is either superior or comparable to other classifiers.

[1]  Senén Barro,et al.  Do we need hundreds of classifiers to solve real world classification problems? , 2014, J. Mach. Learn. Res..

[2]  Xavier Llorà,et al.  Automated alphabet reduction method with evolutionary algorithms for protein structure prediction , 2007, GECCO '07.

[3]  Gilles Venturini,et al.  SIA: A Supervised Inductive Algorithm with Genetic Search for Learning Attributes based Concepts , 1993, ECML.

[4]  Steven Guan,et al.  An incremental approach to genetic-algorithms-based classification , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Stewart W. Wilson.  Knowledge Growth in an Artificial Animal , 1985, ICGA.

[6]  Stewart W. Wilson.  ZCS: A Zeroth Level Classifier System , 1994, Evolutionary Computation.

[7]  Juan M. Corchado,et al.  An evolutionary framework for machine learning applied to medical data , 2019, Knowl. Based Syst..

[8]  Bart Baesens,et al.  To tune or not to tune: rule evaluation for metaheuristic-based sequential covering algorithms , 2013, Data Mining and Knowledge Discovery.

[9]  Alex A. Freitas,et al.  Discovering comprehensible classification rules with a genetic algorithm , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[10]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[11]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[12]  Edwin Lughofer,et al.  Improved fault detection employing hybrid memetic fuzzy modeling and adaptive filters , 2017, Appl. Soft Comput..

[13]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[14]  Jesús S. Aguilar-Ruiz,et al.  Natural Encoding for Evolutionary Supervised Learning , 2007, IEEE Transactions on Evolutionary Computation.

[15]  Miguel Toro,et al.  Evolutionary learning of hierarchical decision rules , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[16]  Francisco Herrera,et al.  Genetics-Based Machine Learning for Rule Induction: State of the Art, Taxonomy, and Comparative Study , 2010, IEEE Transactions on Evolutionary Computation.

[17]  María José del Jesús,et al.  KEEL: a software tool to assess evolutionary algorithms for data mining problems , 2008, Soft Comput..

[18]  Margaret J. Eppstein,et al.  A Tandem Evolutionary Algorithm for Identifying Causal Rules from Complex Data , 2020, Evolutionary Computation.

[19]  Mohammad Razeghi-Jahromi,et al.  Multilabel Classification with Weighted Labels Using Learning Classifier Systems , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[20]  Hisao Ishibuchi,et al.  Selecting fuzzy if-then rules for classification problems using genetic algorithms , 1995, IEEE Trans. Fuzzy Syst..

[21]  Rajib Mall,et al.  Application of elitist multi-objective genetic algorithm for classification rule generation , 2008, Appl. Soft Comput..

[22]  Hisao Ishibuchi,et al.  Hybridization of fuzzy GBML approaches for pattern classification problems , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Ali Karci,et al.  Mining Classification Rules by Using Genetic Algorithms with Non-random Initial Population and Uniform Operator , 2004 .

[24]  Hisao Ishibuchi,et al.  Analysis of interpretability-accuracy tradeoff of fuzzy systems by multiobjective fuzzy genetics-based machine learning , 2007, Int. J. Approx. Reason..

[25]  Sanghamitra Bandyopadhyay,et al.  Multiobjective GAs, quantitative indices, and pattern classification , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Kay Chen Tan,et al.  A coevolutionary algorithm for rules discovery in data mining , 2006, Int. J. Syst. Sci..

[27]  R. Fisher,et al.  STATISTICAL METHODS AND SCIENTIFIC INDUCTION , 1955 .

[28]  Alex A. Freitas,et al.  A survey of evolutionary algorithms for data mining and knowledge discovery , 2003 .

[29]  Hisao Ishibuchi,et al.  Evolutionary Multi-objective Rule Selection for Classification Rule Mining , 2008, Multi-Objective Evolutionary Algorithms for Knowledge Discovery from Databases.

[30]  Ester Bernadó-Mansilla,et al.  Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks , 2003, Evolutionary Computation.

[31]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[32]  Zheng-Zhi Wang,et al.  Center-based nearest neighbor classifier , 2007, Pattern Recognit..

[33]  Filip Rudzinski,et al.  A multi-objective genetic optimization of interpretability-oriented fuzzy rule-based classifiers , 2016, Appl. Soft Comput..

[34]  Sandip Sen,et al.  Using real-valued genetic algorithms to evolve rule sets for classification , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[35]  Hisao Ishibuchi,et al.  Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining , 2004, Fuzzy Sets Syst..

[36]  Jason H. Moore,et al.  An analysis pipeline with statistical and visualization-guided knowledge discovery for Michigan-style learning classifier systems , 2012, IEEE Computational Intelligence Magazine.

[37]  Sanghamitra Bandyopadhyay,et al.  Genetic algorithms for generation of class boundaries , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[38]  Jian Zhuang,et al.  Novel soft subspace clustering with multi-objective evolutionary approach for high-dimensional data , 2013, Pattern Recognit..

[39]  Kenneth DeJong,et al.  Learning with genetic algorithms: An overview , 1988, Machine Learning.

[40]  Ujjwal Maulik,et al.  A Survey of Multiobjective Evolutionary Algorithms for Data Mining: Part I , 2014, IEEE Transactions on Evolutionary Computation.

[41]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[42]  Li-Chen Fu,et al.  A two-phase evolutionary algorithm for multiobjective mining of classification rules , 2010, IEEE Congress on Evolutionary Computation.

[43]  Hisao Ishibuchi,et al.  Modification of Evolutionary Multiobjective Optimization Algorithms for Multiobjective Design of Fuzzy Rule-Based Classification Systems , 2005, The 14th IEEE International Conference on Fuzzy Systems, 2005. FUZZ '05..

[44]  Stephen F. Smith,et al.  Competition-based induction of decision models from examples , 1993, Machine Learning.

[45]  Edmund K. Burke,et al.  Improving the scalability of rule-based evolutionary learning , 2009, Memetic Comput..

[46]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[47]  Francisco Herrera,et al.  A Compact Evolutionary Interval-Valued Fuzzy Rule-Based Classification System for the Modeling and Prediction of Real-World Financial Applications With Imbalanced Data , 2015, IEEE Transactions on Fuzzy Systems.

[48]  Ester Bernadó-Mansilla,et al.  Fuzzy-UCS: A Michigan-Style Learning Fuzzy-Classifier System for Supervised Learning , 2009, IEEE Transactions on Evolutionary Computation.

[49]  Jan Paredis,et al.  Genetic rule induction at an intermediate level , 2002, Knowl. Based Syst..

[50]  Stewart W. Wilson.  Classifier Fitness Based on Accuracy , 1995, Evolutionary Computation.

[51]  Marian B. Gorzalczany,et al.  A multi-objective genetic optimization for fast, fuzzy rule-based credit classification with balanced accuracy and interpretability , 2016, Appl. Soft Comput..

[52]  Kenneth A. De Jong,et al.  Using genetic algorithms for concept learning , 1993, Machine Learning.

[53]  Hideo Tanaka,et al.  Construction of fuzzy classification systems with rectangular fuzzy rules using genetic algorithms , 1994, CVPR 1994.

[54]  Jason H. Moore,et al.  Learning classifier systems: a complete introduction, review, and roadmap , 2009 .

[55]  Humberto Bustince,et al.  Medical diagnosis of cardiovascular diseases using an interval-valued fuzzy rule-based classification system , 2014, Appl. Soft Comput..

[56]  Yaguo Lei,et al.  A multidimensional hybrid intelligent method for gear fault diagnosis , 2010, Expert Syst. Appl..

[57]  Zhang Lei,et al.  A classification rule mining method using hybrid genetic algorithms , 2004, 2004 IEEE Region 10 Conference TENCON 2004..

[58]  Magne Setnes,et al.  GA-fuzzy modeling and classification: complexity and performance , 2000, IEEE Trans. Fuzzy Syst..

[59]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[60]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[61]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[62]  Michael P. Fourman,et al.  Compaction of Symbolic Layout Using Genetic Algorithms , 1985, ICGA.

[63]  Jaume Bacardit,et al.  Bloat Control and Generalization Pressure Using the Minimum Description Length Principle for a Pittsburgh Approach Learning Classifier System , 2005, IWLCS.

[64]  Francisco Herrera,et al.  On the use of evolutionary feature selection for improving fuzzy rough set based prototype selection , 2012, Soft Computing.

[65]  Stewart W. Wilson,et al.  Noname manuscript No. (will be inserted by the editor) Learning Classifier Systems: A Survey , 2022 .

[66]  Cezary Z. Janikow,et al.  A knowledge-intensive genetic algorithm for supervised learning , 1993, Machine Learning.

[67]  Roberto J. Bayardo,et al.  Mining the most interesting rules , 1999, KDD '99.

[68]  Dr. Alex A. Freitas.  Data Mining and Knowledge Discovery with Evolutionary Algorithms , 2002, Natural Computing Series.

[69]  Raju Nedunchezhian,et al.  Mining data streams with concept drifts using genetic algorithm , 2011, Artificial Intelligence Review.

[70]  John H. Holland,et al.  Induction: Processes of Inference, Learning, and Discovery , 1987, IEEE Expert.

[71]  Rajib Mall,et al.  Predictive and comprehensible rule discovery using a multi-objective genetic algorithm , 2006, Knowl. Based Syst..

[72]  Eghbal G. Mansoori,et al.  SGERD: A Steady-State Genetic Algorithm for Extracting Fuzzy Classification Rules From Data , 2008, IEEE Transactions on Fuzzy Systems.

[73]  John H. Holland,et al.  COGNITIVE SYSTEMS BASED ON ADAPTIVE ALGORITHMS1 , 1978 .

[74]  Kalyanmoy Deb,et al.  A survey of evolutionary algorithms using metameric representations , 2019, Genetic Programming and Evolvable Machines.

[75]  Sanghamitra Bandyopadhyay,et al.  VGA-Classifier: design and applications , 2000, IEEE Trans. Syst. Man Cybern. Part B.

[76]  Casimir A. Kulikowski,et al.  Computer Systems That Learn: Classification and Prediction Methods from Statistics, Neural Nets, Machine Learning and Expert Systems , 1990 .

[77]  Saeid Nahavandi,et al.  Classification of healthcare data using genetic fuzzy logic system and wavelets , 2015, Expert Syst. Appl..

[78]  Jaume Bacardit,et al.  Analysis and Improvements of the Adaptive Discretization Intervals Knowledge Representation , 2004, GECCO.

[79]  Victor J. Rayward-Smith,et al.  Developments on a Multi-objective Metaheuristic (MOMH) Algorithm for Finding Interesting Sets of Classification Rules , 2005, EMO.

[80]  Alex Alves Freitas,et al.  A critical review of multi-objective optimization in data mining: a position paper , 2004, SKDD.

[81]  William B. Langdon,et al.  Fitness Causes Bloat in Variable Size Representations , 1997 .

[82]  Mehmet Kaya.  Autonomous classifiers with understandable rule using multi-objective genetic algorithms , 2010, Expert Syst. Appl..

[83]  Victor J. Rayward-Smith,et al.  The application and effectiveness of a multi-objective metaheuristic algorithm for partial classification , 2006, Eur. J. Oper. Res..

[84]  Chaochang Chiu,et al.  A constraint-based genetic algorithm approach for mining classification rules , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[85]  Paramartha Dutta,et al.  A real coded MOGA for mining classification rules with missing attribute values , 2011, ICCCS '11.

[86]  Jaume Bacardit,et al.  Evolving Multiple Discretizations with Adaptive Intervals for a Pittsburgh Rule-Based Learning Classifier System , 2003, GECCO.

[87]  David García,et al.  Overview of the SLAVE learning algorithm: A review of its evolution and prospects , 2014, Int. J. Comput. Intell. Syst..

[88]  Francisco Herrera,et al.  IVTURS: A Linguistic Fuzzy Rule-Based Classification System Based On a New Interval-Valued Fuzzy Reasoning Method With Tuning and Rule Selection , 2013, IEEE Transactions on Fuzzy Systems.

[89]  Alex A. Freitas,et al.  Discovering interesting prediction rules with a genetic algorithm , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[90]  Jaume Bacardit,et al.  GAssist vs. BioHEL: critical assessment of two paradigms of genetics-based machine learning , 2013, Soft Comput..

[91]  Antonio González Muñoz,et al.  Including a simplicity criterion in the selection of the best rule in a genetic fuzzy learning algorithm , 2001, Fuzzy Sets Syst..

[92]  Alex A. Freitas,et al.  A Genetic Algorithm for Generalized Rule Induction , 1999 .

[93]  M. Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .