GEML: A Grammatical Evolution, Machine Learning Approach to Multi-class Classification

In this paper, we propose a hybrid approach to solving multi-class problems which combines evolutionary computation with elements of traditional machine learning. The method, Grammatical Evolution Machine Learning (GEML) adapts machine learning concepts from decision tree learning and clustering methods and integrates these into a Grammatical Evolution framework. We investigate the effectiveness of GEML on several supervised, semi-supervised and unsupervised multi-class problems and demonstrate its competitive performance when compared with several well known machine learning algorithms. The GEML framework evolves human readable solutions which provide an explanation of the logic behind its classification decisions, offering a significant advantage over existing paradigms for unsupervised and semi-supervised learning. In addition we also examine the possibility of improving the performance of the algorithm through the application of several ensemble techniques.

[1]  Ponnuthurai N. Suganthan,et al.  Ensemble Classification and Regression-Recent Developments, Applications and Future Directions [Review Article] , 2016, IEEE Computational Intelligence Magazine.

[2]  Ujjwal Maulik,et al.  Genetic algorithm-based clustering technique , 2000, Pattern Recognit..

[3]  Habib Zaidi,et al.  A novel fuzzy C-means algorithm for unsupervised heterogeneous tumor quantification in PET. , 2010, Medical physics.

[4]  Conor RyanDept,et al.  Automatic Generation of Programs with Grammatical Evolution , 2007 .

[5]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[6]  David B. Fogel What is evolutionary computation , 1995 .

[7]  Flávio Bortolozzi,et al.  Unsupervised feature selection using multi-objective genetic algorithms for handwritten word recognition , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[8]  T Watson Layne,et al.  A Genetic Algorithm Approach to Cluster Analysis , 1998 .

[9]  A. Mojsilovic,et al.  Wavelet image extension for analysis and classification of infarcted myocardial tissue , 1997, IEEE Transactions on Biomedical Engineering.

[10]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[11]  Thomas G. Dietterich Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[12]  Vladan Babovic,et al.  Genetic Programming, Ensemble Methods and the Bias/Variance Tradeoff - Introductory Investigations , 2000, EuroGP.

[13]  Andries Petrus Engelbrecht,et al.  Differential evolution methods for unsupervised image classification , 2005, 2005 IEEE Congress on Evolutionary Computation.

[14]  Ingo Mierswa,et al.  Information preserving multi-objective feature selection for unsupervised learning , 2006, GECCO.

[15]  Anthony Brabazon,et al.  Grammatical Differential Evolution , 2006, IC-AI.

[16]  Ahmed Kattan,et al.  Time-series event-based prediction: An unsupervised learning framework based on genetic programming , 2015, Inf. Sci..

[17]  Réka Albert,et al.  Near linear time algorithm to detect community structures in large-scale networks. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[18]  Anthony Brabazon,et al.  Grammatical Swarm: A Variable-Length Particle Swarm Algorithm , 2006, Swarm Intelligent Systems.

[19]  N. Altman An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression , 1992 .

[20]  Mengjie Zhang,et al.  Parallel Linear Genetic Programming , 2011, EuroGP.

[21]  Derek Greene,et al.  Ensemble clustering in medical diagnostics , 2004, Proceedings. 17th IEEE Symposium on Computer-Based Medical Systems.

[22]  Simone A. Ludwig,et al.  Improving genetic programming classification for binary and multiclass datasets , 2013, 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[23]  Alex A. Freitas,et al.  Evolutionary Computation , 2002 .

[24]  Alex Alves Freitas,et al.  A Survey of Evolutionary Algorithms for Clustering , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[25]  Mark Johnston,et al.  Unsupervised learning for edge detection using Genetic Programming , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[26]  Riccardo Poli,et al.  Unsupervised Problem Decomposition Using Genetic Programming , 2010, EuroGP.

[27]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[28]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[29]  Filippo Menczer,et al.  Feature selection in unsupervised learning via evolutionary search , 2000, KDD '00.

[30]  Anthony Brabazon,et al.  Self-Organizing Swarm (SOSwarm): A Particle Swarm Algorithm for Unsupervised Learning , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[31]  Mengjie Zhang,et al.  Multiclass Object Classification Using Genetic Programming , 2004, EvoWorkshops.

[32]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[33]  Wolfgang Banzhaf,et al.  Evolutionary Computation and Genetic Programming , 2013 .

[34]  Alex Alves Freitas,et al.  A Survey of Evolutionary Algorithms for Decision-Tree Induction , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[35]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[36]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[37]  HaiyanPan,et al.  Genetic Algorithms Applied to Multi-Class Clustering for Gene Expression Data , 2003 .

[38]  Alison A. Motsinger-Reif,et al.  Grammatical evolution decision trees for detecting gene-gene interactions , 2010, BioData Mining.

[39]  Wolfgang Banzhaf,et al.  Rethinking multilevel selection in genetic programming , 2011, GECCO '11.

[40]  Mengjie Zhang,et al.  Using Genetic Programming for Multiclass Classification by Simultaneously Solving Component Binary Classification Problems , 2005, EuroGP.

[41]  Conor Ryan,et al.  GEML: Evolutionary unsupervised and semi-supervised learning of multi-class classification with Grammatical Evolution , 2015, 2015 7th International Joint Conference on Computational Intelligence (IJCCI).

[42]  John R. Koza,et al.  Genetic programming: a paradigm for genetically breeding populations of computer programs to solve problems , 1990 .

[43]  Leonardo Vanneschi,et al.  Land Cover/Land Use Multiclass Classification Using GP with Geometric Semantic Operators , 2013, EvoApplications.

[44]  Conor Ryan,et al.  The Best Things Don't Always Come in Small Packages: Constant Creation in Grammatical Evolution , 2014, EuroGP.

[45]  Mengjie Zhang,et al.  Unsupervised Elimination of Redundant Features Using Genetic Programming , 2009, Australasian Conference on Artificial Intelligence.

[46]  Conor Ryan,et al.  A hybrid approach to the problem of class imbalance , 2013 .

[47]  Mengjie Zhang,et al.  Probability Based Genetic Programming for Multiclass Object Classification , 2004, PRICAI.

[48]  Mark Johnston,et al.  Developing New Fitness Functions in Genetic Programming for Classification With Unbalanced Data , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[49]  Chuanyi Ji,et al.  Combinations of Weak Classifiers , 1996, NIPS.

[50]  Luis Muñoz,et al.  M3GP - Multiclass Classification with GP , 2015, EuroGP.