Cooperative Coevolution of Generalized Multi-Layer Perceptrons

Abstract In this work we present the cooperative coevolution of multi-layer generalized perceptrons. This model is based on the cooperation of different subpopulations of modules, each one being a generalized multi-layered perceptron. In some previous works we have developed a modular cooperative coevolutive model for evolving multi-layer perceptrons with two hidden layers. This model performs very well but tends to generate big networks. In the present paper we show the results of substituting these multi-layer perceptrons by generalized multi-layer perceptrons, which allow a more compact representation of networks. The use of generalized multi-layer perceptrons improved the performance of the evolutionary model with regard to the evolution of other kinds of networks. Another improvement was that the networks obtained were much smaller. The comparison proved statistically significant by means of a Student's t-test.

[1]  Lutz Prechelt,et al.  A quantitative study of experimental evaluations of neural network learning algorithms: Current research practice , 1996, Neural Networks.

[2]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[3]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[4]  Jihoon Yang,et al.  DistAl: an inter-pattern distance-based constructive learning algorithm , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[5]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[6]  David W. Opitz,et al.  Actively Searching for an E(cid:11)ective Neural-Network Ensemble , 1996 .

[7]  Ferdinand Hergert,et al.  Improving model selection by nonconvergent methods , 1993, Neural Networks.

[8]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[9]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[10]  O. Mangasarian,et al.  Robust linear programming discrimination of two linearly inseparable sets , 1992 .

[11]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[12]  Xin Yao,et al.  A Preliminary Study on Designing Artiicial Neural Networks Using Co-evolution , 1995 .

[13]  Nirmal K. Bose,et al.  Neural network design using Voronoi diagrams , 1993, IEEE Trans. Neural Networks.

[14]  D. Fogel.  Evolutionary algorithms in theory and practice , 1997, Complex..

[15]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[16]  Jihoon Yang,et al.  Constructive Neural Network Learning Algorithms , 1996, AAAI/IAAI, Vol. 2.

[17]  J. David Schaffer,et al.  Proceedings of the third international conference on Genetic algorithms , 1989 .

[18]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1992, Artificial Intelligence.

[19]  DistAl: An inter-pattern distance-based constructive learning algorithm , 1999, Intell. Data Anal..

[20]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[21]  Richard S. Johannes,et al.  Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus , 1988 .

[22]  César Hervás-Martínez,et al.  COVNET: a cooperative coevolutionary model for evolving artificial neural networks , 2003, IEEE Trans. Neural Networks.

[23]  M. Goldstein,et al.  Multivariate Analysis: Methods and Applications , 1984 .

[24]  Vasant G Honavar,et al.  MTiling A Constructive Neural Network Learning Algorithm for Multi Category Pattern Classi cation , 1996 .

[25]  Kalyanmoy Deb,et al.  MULTI-OBJECTIVE FUNCTION OPTIMIZATION USING NON-DOMINATED SORTING GENETIC ALGORITHMS , 1994 .

[26]  OpitzDavid,et al.  Popular ensemble methods , 1999 .

[27]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[28]  Sun-Yuan Kung,et al.  Digital neural networks , 1993, Prentice Hall Information and System Sciences Series.

[29]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[30]  David B. Fogel,et al.  Evolving artificial intelligence , 1992 .

[31]  Peter J. Fleming,et al.  Genetic Algorithms for Multiobjective Optimization: FormulationDiscussion and Generalization , 1993, ICGA.

[32]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Darrell Whitley,et al.  Genitor: a different genetic algorithm , 1988 .

[34]  Thomas Bäck,et al.  Evolutionary Algorithms in Theory and Practice , 1996 .

[35]  Wolfgang Banzhaf,et al.  A comparison of linear genetic programming and neural networks in medical data mining , 2001, IEEE Trans. Evol. Comput..

[36]  Asim Roy,et al.  An algorithm to generate radial basis function (RBF)-like nets for classification problems , 1995, Neural Networks.

[37]  C. Fonseca,et al.  GENETIC ALGORITHMS FOR MULTI-OBJECTIVE OPTIMIZATION: FORMULATION, DISCUSSION, AND GENERALIZATION , 1993 .

[38]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[39]  Terry Caelli,et al.  Model-based neural networks , 1993, Neural Networks.

[40]  Murali S. Shanker,et al.  Using Neural Networks To Predict the Onset of Diabetes Mellitus , 1996, J. Chem. Inf. Comput. Sci..

[41]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[42]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[43]  R. Detrano,et al.  International application of a new probability algorithm for the diagnosis of coronary artery disease. , 1989, The American journal of cardiology.

[44]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[45]  Athanassios Siapas,et al.  Criticality and Parallelism in Combinatorial Optimization , 1996, Science.

[46]  Leo Breiman,et al.  Stacked regressions , 2004, Machine Learning.

[47]  Michael I. Jordan,et al.  Hierarchies of Adaptive Experts , 1991, NIPS.

[48]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[49]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[50]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[51]  Mitchell A. Potter,et al.  The design and analysis of a computational model of cooperative coevolution , 1997 .

[52]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[53]  Michael Conrad,et al.  Combining evolution with credit apportionment: A new learning algorithm for neural nets , 1994, Neural Networks.

[54]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[55]  Kalyanmoy Deb,et al.  An Investigation of Niche and Species Formation in Genetic Function Optimization , 1989, ICGA.

[56]  Ling Guan,et al.  Modularity in neural computing , 1999, Proc. IEEE.

[57]  L. Breiman.  Arcing classifier (with discussion and a rejoinder by the author) , 1998 .

[58]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[59]  Robert I. Damper,et al.  On neural-network implementations of k-nearest neighbor pattern classifiers , 1997 .

[60]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[61]  Heggere S. Ranganath,et al.  Self partitioning neural networks for target recognition , 1995, Neural Networks.

[62]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[63]  Jordan B. Pollack,et al.  Pareto Optimality in Coevolutionary Learning , 2001, ECAL.

[64]  César Hervás-Martínez,et al.  Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks) , 2002, Neural Networks.

[65]  Eckart Zitzler,et al.  Evolutionary algorithms for multiobjective optimization: methods and applications , 1999 .

[66]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[67]  Xin Yao,et al.  Simultaneous training of negatively correlated neural networks in an ensemble , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[68]  Kalyanmoy Deb,et al.  Evolutionary Algorithms for Multi-Criterion Optimization in Engineering Design , 1999 .

[69]  Simon Haykin,et al.  Neural networks expand SP's horizons , 1996, IEEE Signal Process. Mag..

[70]  Huan Liu,et al.  Learning a Neural Tree , 1992 .

[71]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[72]  Tamás D. Gedeon,et al.  Exploring constructive cascade networks , 1999, IEEE Trans. Neural Networks.