Comparing evolutionary hybrid systems for design and optimization of multilayer perceptron structure along training parameters

In this paper we present a comparative study of several methods that combine evolutionary algorithms and local search to optimize multilayer perceptrons: A method that optimizes the architecture and initial weights of multilayer perceptrons; another that searches for training algorithm parameters, and finally, a co-evolutionary algorithm, introduced here, that handles the architecture, the network's initial weights and the training algorithm parameters. Our aim is to determine how the co-evolutive method can obtain better results from the point of view of running time and classification ability. Experimental results show that the co-evolutionary method obtains similar or better results than the other approaches, requiring far less training epochs and thus, reducing running time.

[1]  W. Daniel Hillis,et al.  Co-evolving parasites improve simulated evolution as an optimization procedure , 1990 .

[2]  Xin Yao,et al.  Recent Advances in Evolutionary Computation , 2006, Journal of Computer Science and Technology.

[3]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[4]  H. White,et al.  There exists a neural network that does not make avoidable mistakes , 1988, IEEE 1988 International Conference on Neural Networks.

[5]  César Hervás-Martínez,et al.  COVNET: a cooperative coevolutionary model for evolving artificial neural networks , 2003, IEEE Trans. Neural Networks.

[6]  Vasant Honavar,et al.  Advances in the Evolutionary Synthesis of Intelligent Agents , 2001 .

[7]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[8]  Jihoon Yang,et al.  Constructive Neural-Network Learning Algorithms for Pattern Classification , 2000 .

[9]  Charles E. Taylor,et al.  Artificial Life II , 1991 .

[10]  Jihoon Yang,et al.  Pruning strategies for the MTiling constructive learning algorithm , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[11]  Werner Kinnebrock,et al.  Accelerating the standard backpropagation method using a genetic approach , 1994, Neurocomputing.

[12]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[13]  Juan Julián Merelo Guervós,et al.  Optimisation of Multilayer Perceptrons Using a Distributed Evolutionary Algorithm with SOAP , 2002, PPSN.

[14]  A. Gray,et al.  I. THE ORIGIN OF SPECIES BY MEANS OF NATURAL SELECTION , 1963 .

[15]  N. García-Pedrajas,et al.  SYMBIONT: a cooperative evolutionary model for evolving artificial neural networks for classification , 2002 .

[16]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[17]  Frédéric Gruau,et al.  Genetic synthesis of Boolean neural networks with a cell rewriting developmental process , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[18]  Richard K. Belew,et al.  New Methods for Competitive Coevolution , 1997, Evolutionary Computation.

[19]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning through Symbiotic Evolution , 2004 .

[20]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[21]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[22]  D. M. Hutton,et al.  Advances in the Evolutionary Synthesis of Intelligent Agents , 2002 .

[23]  Jihoon Yang,et al.  MUpstart-a constructive neural network learning algorithm for multi-category pattern classification , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[24]  Rudolf F. Albrecht,et al.  Artificial Neural Nets and Genetic Algorithms , 1995, Springer Vienna.

[25]  Rajesh Parekh,et al.  Constructive theory refinement in knowledge based neural networks , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[26]  Xin Yao,et al.  Evolving hybrid ensembles of learning machines for better generalisation , 2006, Neurocomputing.

[27]  Qiangfu Zhao.  Co-evolutionary learning of neural networks , 1998, J. Intell. Fuzzy Syst..

[28]  Juan Julián Merelo Guervós,et al.  G-lvq, a Combination of Genetic Algorithms and Lvq , 1995, ICANNGA.

[29]  Juan Julián Merelo Guervós,et al.  SA-Prop: Optimization of Multilayer Perceptron Parameters Using Simulated Annealing , 1999, IWANN.

[30]  Phil Husbands,et al.  Simulated Co-Evolution as the Mechanism for Emergent Planning and Scheduling , 1991, ICGA.

[31]  Xin Yao,et al.  Evolutionary framework for the construction of diverse hybrid ensembles , 2005, ESANN.

[32]  Juan Julián Merelo Guervós,et al.  Specifying Evolutionary Algorithms in XML , 2003, IWANN.

[33]  Dario Floreano,et al.  Neuroevolution with Analog Genetic Encoding , 2006, PPSN.

[34]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[35]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[36]  César Hervás-Martínez,et al.  Cooperative coevolution of artificial neural network ensembles for pattern classification , 2005, IEEE Transactions on Evolutionary Computation.

[37]  J. J. Merelo.  Optimization of Classifiers Using Genetic Algorithms , 1996 .

[38]  Juan Julián Merelo Guervós,et al.  Diseño de Redes Neuronales Artificiales mediante Algoritmos Evolutivos , 2001, Inteligencia Artif..

[39]  Carlos A. Reyes García,et al.  ARGEN + AREPO: Improving the Search Process with Artificial Genetic Engineering , 2005, IWANN.

[40]  Ethem Alpaydin,et al.  GAL: Networks That Grow When They Learn and Shrink When They Forget , 1994, Int. J. Pattern Recognit. Artif. Intell..

[41]  Xin Yao,et al.  Experimental study on population-based incremental learning algorithms for dynamic optimization problems , 2005, Soft Comput..

[42]  Jan Paredis,et al.  The Symbiotic Evolution of Solutions and Their Representations , 1995, International Conference on Genetic Algorithms.

[43]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[44]  Johan A. K. Suykens,et al.  Genetic Weight Optimization of a Feedforward Neural Network Controller , 1993 .

[45]  Michael Conrad,et al.  Combining evolution with credit apportionment: A new learning algorithm for neural nets , 1994, Neural Networks.

[46]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[47]  Björn Olsson.  Co-evolutionary search in asymmetric spaces , 2001, Inf. Sci..

[48]  Geoffrey E. Hinton,et al.  Proceedings of the 1988 Connectionist Models Summer School , 1989 .

[49]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[50]  Juan Julián Merelo Guervós,et al.  G-Prop: Global optimization of multilayer perceptrons using GAs , 2000, Neurocomputing.

[51]  Zbigniew Michalewicz,et al.  Genetic algorithms + data structures = evolution programs (3rd ed.) , 1996 .

[52]  A. Cangelosi,et al.  Cell division and migration in a 'genotype' for neural networks (Cell division and migration in neural networks) , 1993 .

[53]  Gérard Dreyfus,et al.  Toward a Principled Methodology for Neural Network Design and Performance Evaluation in QSAR. Application to the Prediction of LogP , 1998, J. Chem. Inf. Comput. Sci..

[54]  Chulhyun Kim,et al.  Forecasting time series with genetic fuzzy predictor ensemble , 1997, IEEE Trans. Fuzzy Syst..

[55]  O. Mangasarian,et al.  Pattern Recognition Via Linear Programming: Theory and Application to Medical Diagnosis , 1989 .

[56]  Thomas F. Coleman,et al.  Large-Scale Numerical Optimization , 1990 .

[57]  Ignacio Bellido,et al.  Backpropagation Growing Networks: Towards Local Minima Elimination , 1991, IWANN.

[58]  Vassilios Petridis,et al.  A hybrid genetic algorithm for training neural networks , 1992 .

[59]  C. Jutten,et al.  Gal: Networks That Grow When They Learn and Shrink When They Forget , 1991 .

[60]  Jenq-Neng Hwang,et al.  The cascade-correlation learning: a projection pursuit learning perspective , 1996, IEEE Trans. Neural Networks.

[61]  Jiwen Dong,et al.  Time-series forecasting using flexible neural tree model , 2005, Inf. Sci..

[62]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[63]  Rajkumar Roy,et al.  Advances in Soft Computing , 2018, Lecture Notes in Computer Science.

[64]  John J. Grefenstette,et al.  Optimization of Control Parameters for Genetic Algorithms , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[65]  J. David Schaffer,et al.  Proceedings of the third international conference on Genetic algorithms , 1989 .

[66]  Marko Gronroos,et al.  Evolutionary Design of Neural Networks , 1998 .

[67]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[68]  César Hervás-Martínez,et al.  Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks) , 2002, Neural Networks.

[69]  Jan Paredis,et al.  Coevolutionary computation , 1995 .

[70]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .

[71]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.

[72]  Dimitrios Gunopulos,et al.  Adaptive metric nearest neighbor classification , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[73]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[74]  Juan Julián Merelo Guervós,et al.  Evolving Multilayer Perceptrons , 2000, Neural Processing Letters.

[75]  Pedro Ángel Castillo Valdivieso,et al.  G-Prop-II: global optimization of multilayer perceptrons using GAs , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[76]  R. Lippmann,et al.  An introduction to computing with neural nets , 1987, IEEE ASSP Magazine.

[77]  Ignacio Rojas,et al.  Statistical analysis of the parameters of a neuro-genetic algorithm , 2002, IEEE Trans. Neural Networks.

[78]  Nicolás García-Pedrajas,et al.  A cooperative constructive method for neural networks for pattern recognition , 2007, Pattern Recognit..

[79]  Chou-Yuan Lee,et al.  A hybrid search algorithm with heuristics for resource allocation problem , 2005, Inf. Sci..

[80]  Chandrika Kamath,et al.  Inducing oblique decision trees with evolutionary algorithms , 2003, IEEE Trans. Evol. Comput..

[81]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[82]  Anna Maria Fanelli,et al.  A Method of Pruning Layered Feed-Forward Neural Networks , 1993, IWANN.

[83]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Pattern Classification , 1995 .

[84]  Michael Georgiopoulos,et al.  Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization , 1997, Neurocomputing.

[85]  Juan Julián Merelo Guervós,et al.  G-Prop-III: Global Optimization of Multilayer Perceptrons using an Evolutionary Algorithm , 1999, GECCO.

[86]  Hean-Lee Poh,et al.  Analysis of Pruning in Backpropagation Networks for Artificial and Real Worls Mapping Problems , 1995, IWANN.

[87]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[88]  Rory A. Fisher,et al.  THE COMPARISON OF SAMPLES WITH POSSIBLY UNEQUAL VARIANCES , 1939 .

[89]  Rory A. Fisher,et al.  Theory of Statistical Estimation , 1925, Mathematical Proceedings of the Cambridge Philosophical Society.

[90]  Juan Julián Merelo Guervós,et al.  Optimization of a Competitive Learning Neural Network by Genetic Algorithms , 1993, IWANN.

[91]  S Usui,et al.  Robustness, evolvability, and optimality of evolutionary neural networks. , 2005, Bio Systems.

[92]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[93]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[94]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[95]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[96]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[97]  Ivanoe De Falco,et al.  Evolutionary Neural Networks for Nonlinear Dynamics Modeling , 1998, PPSN.

[98]  Paul T. Jackway,et al.  Co-operative Evolution of a Neural Classifier and Feature Subset , 1998, SEAL.

[99]  Bernard Zenko,et al.  Is Combining Classifiers with Stacking Better than Selecting the Best One? , 2004, Machine Learning.

[100]  A. E. Eiben,et al.  Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[101]  C. Darwin.  The Origin of Species by Means of Natural Selection, Or, The Preservation of Favoured Races in the Struggle for Life , 1859 .

[102]  Risto Miikkulainen,et al.  Hierarchical evolution of neural networks , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[103]  Michele Marchesi,et al.  A hybrid genetic-neural architecture for stock indexes forecasting , 2005, Inf. Sci..

[104]  Ilona Jagielska,et al.  An investigation into the application of neural networks, fuzzy logic, genetic algorithms, and rough sets to automated knowledge acquisition for classification problems , 1999, Neurocomputing.

[105]  K. Saito,et al.  Cooperative co-evolutionary algorithm-how to evaluate a module? , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[106]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[107]  Lashon B. Booker,et al.  Proceedings of the fourth international conference on Genetic algorithms , 1991 .

[108]  Phil Husbands,et al.  Distributed Coevolutionary Genetic Algorithms for Multi-Criteria and Multi-Constraint Optimisation , 1994, Evolutionary Computing, AISB Workshop.

[109]  David G. Stork,et al.  Evolution and Learning in Neural Networks: The Number and Distribution of Learning Trials Affect the Rate of Evolution , 1990, NIPS 1990.

[110]  Vasant Honavar,et al.  Optimization of Classifiers Using Genetic Algorithms , 2001 .

[111]  Reinhold Huber,et al.  Evolving Topologies of Artificial Neural Networks Adapted to Image Processing Tasks , 1996 .

[112]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[113]  César Hervás-Martínez,et al.  Cascade Ensembles , 2005, IWANN.