An analysis of accuracy-diversity trade-off for hybrid combined system with multiobjective predictor selection

This study examines the contribution of diversity under a multi-objective context for the promotion of learners in an evolutionary system that generates combinations of partially trained learners. The examined system uses a grammar-driven genetic programming to evolve hierarchical, multi-component combinations of multilayer perceptrons and support vector machines for regression. Two advances are studied. First, a ranking formula is developed for the selection probability of the base learners. This formula incorporates both a diversity measure and the performance of learners, and it is tried over a series of artificial and real-world problems. Results show that when the diversity of a learner is incorporated with equal weights to the learner performance in the evolutionary selection process, the system is able to provide statistically significantly better generalization. The second advance examined is a substitution phase for learners that are over-dominated, under a multi-objective Pareto domination assessment scheme. Results here show that the substitution does not improve significantly the system performance, thus the exclusion of very weak learners, is not a compelling task for the examined framework.

[1]  C. Fonseca,et al.  GENETIC ALGORITHMS FOR MULTI-OBJECTIVE OPTIMIZATION: FORMULATION, DISCUSSION, AND GENERALIZATION , 1993 .

[2]  Sung-Bae Cho,et al.  The classification of cancer based on DNA microarray data that uses diverse ensemble genetic programming , 2006, Artif. Intell. Medicine.

[3]  Samy Bengio,et al.  SVMTorch: Support Vector Machines for Large-Scale Regression Problems , 2001, J. Mach. Learn. Res..

[4]  David J. Montana,et al.  Strongly Typed Genetic Programming , 1995, Evolutionary Computation.

[5]  Kagan Tumer,et al.  Error Correlation and Error Reduction in Ensemble Classifiers , 1996, Connect. Sci..

[6]  Giandomenico Spezzano,et al.  Ensemble Techniques for Parallel Genetic Programming Based Classifiers , 2003, EuroGP.

[7]  Leonardo Vanneschi,et al.  Diversity analysis in cellular and multipopulation genetic programming , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[8]  Francesco Gagliardi,et al.  A novel grammar-based genetic programming approach to clustering , 2005, SAC '05.

[9]  J. K. Kinnear,et al.  Advances in Genetic Programming , 1994 .

[10]  Jianxin Wu,et al.  Genetic Algorithm based Selective Neural Network Ensemble , 2001, IJCAI.

[11]  Leonardo Vanneschi,et al.  An Empirical Study of Multipopulation Genetic Programming , 2003, Genetic Programming and Evolvable Machines.

[12]  Eckart Zitzler,et al.  Handling Uncertainty in Indicator-Based Multiobjective Optimization , 2006 .

[13]  Bogdan Gabrys,et al.  Classifier selection for majority voting , 2005, Inf. Fusion.

[14]  Peter Stone,et al.  Polynomial Regression with Automated Degree: A Function Approximator for Autonomous Agents , 2006, 2006 18th IEEE International Conference on Tools with Artificial Intelligence (ICTAI'06).

[15]  Bernard Manderick,et al.  Fine-Grained Parallel Genetic Algorithms , 1989, ICGA.

[16]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[17]  Athanasios Tsakonas,et al.  GRADIENT: Grammar-driven genetic programming framework for building multi-component, hierarchical predictive systems , 2012, Expert Syst. Appl..

[18]  Rolf Drechsler,et al.  Multi-objective Optimisation Based on Relation Favour , 2001, EMO.

[19]  André L. V. Coelho,et al.  Multi-objective design of hierarchical consensus functions for clustering ensembles via genetic programming , 2011, Decis. Support Syst..

[20]  Wolfgang Banzhaf,et al.  More on Computational Effort Statistics for Genetic Programming , 2003, EuroGP.

[21]  Brijesh Verma,et al.  Selection and impact of different topologies in multi-layered hierarchical fuzzy systems , 2011, Applied Intelligence.

[22]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[23]  María Dolores Rodríguez-Moreno,et al.  An empirical study on the accuracy of computational effort in Genetic Programming , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[24]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[25]  Robert A. Jacobs,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1993, Neural Computation.

[26]  R. Detrano,et al.  International application of a new probability algorithm for the diagnosis of coronary artery disease. , 1989, The American journal of cardiology.

[27]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[28]  Reiko Tanese,et al.  Distributed Genetic Algorithms , 1989, ICGA.

[29]  Franz Oppacher,et al.  An Analysis of Koza's Computational Effort Statistic for Genetic Programming , 2002, EuroGP.

[30]  Qingfu Zhang,et al.  Multiobjective evolutionary algorithms: A survey of the state of the art , 2011, Swarm Evol. Comput..

[31]  P. K. Chawdhry,et al.  Soft Computing in Engineering Design and Manufacturing , 1998, Springer London.

[32]  Peter J. Fleming,et al.  Genetic Algorithms for Multiobjective Optimization: FormulationDiscussion and Generalization , 1993, ICGA.

[33]  Witold Pedrycz,et al.  A new selective neural network ensemble with negative correlation , 2012, Applied Intelligence.

[34]  J.Ma Troya Linero,et al.  Evolutionary design of fuzzy logic controllers using strongly-typed GP , 1999 .

[35]  Rajeev Kumar,et al.  Improved Sampling of the Pareto-Front in Multiobjective Genetic Optimizations by Steady-State Evolution: A Pareto Converging Genetic Algorithm , 2002, Evolutionary Computation.

[36]  David W. Opitz,et al.  Generating Accurate and Diverse Members of a Neural-Network Ensemble , 1995, NIPS.

[37]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[38]  Lawrence O. Hall,et al.  Ensemble diversity measures and their application to thinning , 2004, Inf. Fusion.

[39]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[40]  Robert A. Jacobs,et al.  Bias/Variance Analyses of Mixtures-of-Experts Architectures , 1997, Neural Computation.

[41]  Soon-Thiam Khu,et al.  An Investigation on Preference Order Ranking Scheme for Multiobjective Evolutionary Optimization , 2007, IEEE Transactions on Evolutionary Computation.

[42]  Francisco Herrera,et al.  A multi-objective evolutionary algorithm for an effective tuning of fuzzy logic controllers in heating, ventilating and air conditioning systems , 2012, Applied Intelligence.

[43]  Siddhartha Bhattacharyya,et al.  Genetic programming in classifying large-scale data: an ensemble method , 2004, Inf. Sci..

[44]  Ludmila I. Kuncheva,et al.  Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy , 2003, Machine Learning.

[45]  Bogdan Gabrys,et al.  New Measure of Classifier Dependency in Multiple Classifier Systems , 2002, Multiple Classifier Systems.

[46]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[47]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[48]  David W. Aha,et al.  Instance‐based prediction of real‐valued attributes , 1989, Comput. Intell..

[49]  Eckart Zitzler,et al.  Indicator-Based Selection in Multiobjective Search , 2004, PPSN.

[50]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[51]  John R. Koza,et al.  Parallel genetic programming: a scalable implementation using the transputer network architecture , 1996 .

[52]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[53]  Ron Kohavi,et al.  Bias Plus Variance Decomposition for Zero-One Loss Functions , 1996, ICML.

[54]  Bogdan Gabrys,et al.  Learning hybrid neuro-fuzzy classifier models from data: to combine or not to combine? , 2004, Fuzzy Sets Syst..

[55]  I. Yeh.  Modeling slump of concrete with fly ash and superplasticizer , 2008 .

[56]  Marco Laumanns,et al.  Evolutionary Multiobjective Design in Automotive Development , 2005, Applied Intelligence.

[57]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[58]  Peter J. Bentley,et al.  Finding Acceptable Solutions in the Pareto-Optimal Range using Multiobjective Genetic Algorithms , 1998 .

[59]  L. Cooper,et al.  When Networks Disagree: Ensemble Methods for Hybrid Neural Networks , 1992 .

[60]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[61]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[62]  Bogdan Gabrys,et al.  Ridge regression ensemble for toxicity prediction , 2010, ICCS.

[63]  Marco Laumanns,et al.  SPEA2: Improving the Strength Pareto Evolutionary Algorithm For Multiobjective Optimization , 2002 .