Generation and simplification of Artificial Neural Networks by means of Genetic Programming

The development of Artificial Neural Networks (ANNs) is traditionally a slow process in which human experts are needed to experiment on different architectural procedures until they find the one that presents the correct results that solve a specific problem. This work describes a new technique that uses Genetic Programming (GP) in order to automatically develop simple ANNs, with a low number of neurons and connections. Experiments have been carried out in order to measure the behavior of the system and also to compare the results obtained using other ANN generation and training methods with evolutionary computation (EC) tools. The obtained results are, in the worst case, at least comparable to existing techniques and, in many cases, substantially better. As explained herein, the system has other important features such as variable discrimination, which provides new information on the problems to be solved.

[1]  Garrison W. Greenwood.  Training partially recurrent neural networks using evolutionary strategies , 1997, IEEE Trans. Speech Audio Process..

[2]  Nichael Lynn Cramer,et al.  A Representation for the Adaptive Generation of Simple Sequential Programs , 1985, ICGA.

[3]  John R. Koza,et al.  Hierarchical Genetic Algorithms Operating on Populations of Computer Programs , 1989, IJCAI.

[4]  E. Thorndike.  On the Organization of Intellect. , 1921 .

[5]  Teresa Bernarda Ludermir,et al.  A multi-objective memetic and hybrid methodology for optimizing the parameters and performance of artificial neural networks , 2010, Neurocomputing.

[6]  Lakhmi C. Jain,et al.  Using genetic algorithms with grammar encoding to generate neural networks , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[7]  Daniel Rivero,et al.  Using Genetic Programming for Character Discrimination in Damaged Documents , 2004, EvoWorkshops.

[8]  Filippo Menczer,et al.  Evidence of hyperplanes in the genetic learning of neural networks , 2004, Biological Cybernetics.

[9]  D. Parisi,et al.  Evolution and learning in neural networks , 2002 .

[10]  A. Cangelosi,et al.  Cell division and migration in a 'genotype' for neural networks (Cell division and migration in neural networks) , 1993 .

[11]  John Dickinson,et al.  Using the Genetic Algorithm to Generate LISP Source Code to Solve the Prisoner's Dilemma , 1987, ICGA.

[12]  Juan R. Rabuñal,et al.  Aceleración del entrenamiento de Redes de Neuronas Artificiales mediante Algoritmos Genéticos utilizando la GPU , 2007 .

[13]  Shimon Whiteson,et al.  Stochastic Optimization for Collision Selection in High Energy Physics , 2006, AAAI.

[14]  Bill C White,et al.  Optimization of neural network architecture using genetic programming improves detection and modeling of gene-gene interactions in studies of human diseases , 2003, BMC Bioinformatics.

[15]  Lutz Prechelt,et al.  Early Stopping - But When? , 2012, Neural Networks: Tricks of the Trade.

[16]  Jaehong Park,et al.  Evolutionary projection neural networks , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[17]  Michael A. Arbib,et al.  The handbook of brain theory and neural networks , 1995, A Bradford book.

[18]  Daniel Rivero,et al.  Time Series Forecast with Anticipation Using Genetic Programming , 2005, IWANN.

[19]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[20]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[21]  Daniel Rivero,et al.  Artificial Neural Network Development by means of Genetic Programming with Graph Codification , 2008 .

[22]  H. de Garis,et al.  GenNets: genetically programmed neural nets-using the genetic algorithm to train neural nets whose inputs and/or outputs vary in time , 1991 .

[23]  Daniel Rivero,et al.  Prediction and modeling of the rainfall-runoff transformation of a typical urban basin using ann and gp , 2003, Appl. Artif. Intell..

[24]  Kyu Ho Park,et al.  Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates , 1996, Neurocomputing.

[25]  Daniel Rivero,et al.  A New Approach to the Extraction of ANN Rules and to Their Generalization Capacity Through GP , 2004, Neural Computation.

[26]  Kalyanmoy Deb,et al.  A Computationally Efficient Evolutionary Algorithm for Real-Parameter Optimization , 2002, Evolutionary Computation.

[27]  Lazaros S. Iliadis,et al.  Feature extraction for time-series data: An artificial neural network evolutionary training model for the management of mountainous watersheds , 2009, Neurocomputing.

[28]  Riccardo Poli,et al.  On Two Approaches to Image Processing Algorithm Design for Binary Images Using GP , 2003, EvoWorkshops.

[29]  Richard M. Friedberg,et al.  A Learning Machine: Part I , 1958, IBM J. Res. Dev..

[30]  John R. Koza,et al.  Genetic generation of both the weights and architecture for a neural network , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[31]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[32]  Hussein A. Abbass,et al.  Data Mining: A Heuristic Approach , 2002 .

[33]  Daniel Rivero,et al.  Using genetic programming for artificial neural network development and simplification , 2006 .

[34]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[35]  Julián Dorado.  Modelo de un sistema para la selección automática en dominios complejos con una estrategia cooperativa, de conjuntos de entrenamiento y arquitecturas ideales de redes de neuronas artificiales ulilizando alogaritmos genéticos , 2011 .

[36]  N. Dodd,et al.  Optimisation of artificial neural network structure using genetic techniques on multiple transputers , 1991 .

[37]  Wlodzislaw Duch,et al.  A new methodology of extraction, optimization and application of crisp and fuzzy logical rules , 2001, IEEE Trans. Neural Networks.

[38]  T. Kathirvalavakumar,et al.  Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks , 2009, Neurocomputing.

[39]  Lalit M. Patnaik,et al.  Learning neural network weights using genetic algorithms-improving performance by search-space reduction , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[40]  I. Jolliffe.  Principal Component Analysis , 2002 .

[41]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[42]  E C Wasson,et al.  A step toward computer-assisted mammography using evolutionary programming and neural networks. , 1997, Cancer letters.

[43]  Dirk Thierens,et al.  Non-redundant genetic coding of neural networks , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[44]  L. Cooper,et al.  When Networks Disagree: Ensemble Methods for Hybrid Neural Networks , 1992 .

[45]  Roberto Raul Kampfner.  Computational modeling of evolutionary learning , 1981 .

[46]  Francisco Herrera,et al.  Un estudio empírico preliminar sobre los tests estadísticos más habituales en el aprendizaje automático , 2004 .

[47]  Xin Yao,et al.  Evolving hybrid ensembles of learning machines for better generalisation , 2006, Neurocomputing.

[48]  Pedro A. Castillo,et al.  Artificial Neural Networks Design using Evolutionary Algorithms , 2003 .

[49]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[50]  Yves Lecourtier,et al.  Optimizing a Neural Network Architecture with an Adaptive Parameter Genetic Algorithm , 1997, IWANN.

[51]  R R Kampfner,et al.  Computational modeling of evolutionary learning processes in the brain. , 1983, Bulletin of mathematical biology.

[52]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[53]  Robert F. Port,et al.  Fractally configured neural networks , 1991, Neural Networks.

[54]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[55]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[56]  Xin Yao,et al.  Evolutionary design of artificial neural networks with different nodes , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[57]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[58]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[59]  Minrui Fei,et al.  A fast multi-output RBF neural network construction method , 2010, Neurocomputing.

[60]  Nicolas Thierry-Mieg,et al.  A new pooling strategy for high-throughput screening: the Shifted Transversal Design , 2006, BMC Bioinformatics.

[61]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[62]  Xin Yao,et al.  Co-evolutionary modular neural networks for automatic problem decomposition , 2005, 2005 IEEE Congress on Evolutionary Computation.

[63]  Frédéric Gruau,et al.  Automatic Definition of Modular Neural Networks , 1994, Adapt. Behav..

[64]  Robert F. Harrison,et al.  Optimization and training of feedforward neural networks by genetic algorithms , 1991 .

[65]  Tariq Samad,et al.  Designing Application-Specific Neural Networks Using the Genetic Algorithm , 1989, NIPS.

[66]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[67]  Daniel Rivero,et al.  Prediction and Modelling of the Flow of a Typical Urban Basin through Genetic Programming , 2002, EvoWorkshops.

[68]  Stefano Nolfi,et al.  Evolutionary Robotics: The Biology, Intelligence, and Technology of Self-Organizing Machines , 2000 .

[69]  T. M. English.  Proceedings of the third annual conference on evolutionary programming: A.V. Sebald and L.J. Fogel, River Edge, NJ: World Scientific, ISBN 981-02-1810-9, 371 pages, hardbound, $78 , 1995 .

[70]  Giles Mayley,et al.  Landscapes, Learning Costs, and Genetic Assimilation , 1996, Evolutionary Computation.

[71]  Yoshua Bengio,et al.  On the Optimization of a Synaptic Learning Rule , 2007 .

[72]  Enrique Alba,et al.  Full Automatic ANN Design: A Genetic Approach , 1993, IWANN.

[73]  Jonathan Baxter.  The evolution of learning algorithms for artificial neural networks , 1993 .

[74]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[75]  Devesh Patel.  Using genetic algorithms to construct a network for financial prediction , 1996, Electronic Imaging.

[76]  James F. Frenzel,et al.  Training product unit neural networks with genetic algorithms , 1993, IEEE Expert.

[77]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[78]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[79]  Wei Yan,et al.  A hybrid genetic/BP algorithm and its application for radar target classification , 1997, Proceedings of the IEEE 1997 National Aerospace and Electronics Conference. NAECON 1997.

[80]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[81]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[82]  Ah Chung Tsoi,et al.  A Constructive Algorithm for the Training of a Multilayer Perceptron Based on the Genetic Algorithm , 1993, Complex Syst..

[83]  M. Stone.  Cross-validation:a review 2 , 1978 .

[84]  César Hervás-Martínez,et al.  Cooperative coevolution of artificial neural network ensembles for pattern classification , 2005, IEEE Transactions on Evolutionary Computation.

[85]  David J. Montana,et al.  Strongly Typed Genetic Programming , 1995, Evolutionary Computation.

[86]  James L. McClelland,et al.  James L. McClelland, David Rumelhart and the PDP Research Group, Parallel distributed processing: explorations in the microstructure of cognition . Vol. 1. Foundations . Vol. 2. Psychological and biological models . Cambridge MA: M.I.T. Press, 1987. , 1989, Journal of Child Language.

[87]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008 .

[88]  Jihoon Yang,et al.  Feature Subset Selection Using a Genetic Algorithm , 1998, IEEE Intell. Syst..

[89]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[90]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[91]  G. Schnitger,et al.  Efficient Approximation with Neural Networks: A Comparison of Gate Functions , 2005 .

[92]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning through Symbiotic Evolution , 2004 .

[93]  Panos A. Ligomenides,et al.  GANNET: a genetic algorithm for searching topology and weight spaces in neural network design. The first step in finding a neural network solution , 1993 .

[94]  Robert E. Smith,et al.  Is a Learning Classifier System a Type of Neural Network? , 1994, Evolutionary Computation.

[95]  Marco Tomassini,et al.  Evolutionary Design of Artificial Neural Networks , 2001 .

[96]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .

[97]  Daniel Rivero,et al.  Automatic Design of ANNs by Means of GP for Data Mining Tasks: Iris Flower Classification Problem , 2007, ICANNGA.

[98]  D B Fogel,et al.  Evolving neural networks for detecting breast cancer. , 1995, Cancer letters.

[99]  Xin Yao,et al.  Multi-network evolutionary systems and automatic decomposition of complex problems , 2006, Int. J. Gen. Syst..

[100]  Yoshua Bengio,et al.  Learning a synaptic learning rule , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[101]  Juan R. Rabuñal,et al.  Artificial Neural Networks in Real-Life Applications , 2005 .

[102]  Kristin P. Bennett,et al.  Feature selection for in-silico drug design using genetic algorithms and neural networks , 2001, SMCia/01. Proceedings of the 2001 IEEE Mountain Workshop on Soft Computing in Industrial Applications (Cat. No.01EX504).

[103]  Stefano Cagnoni,et al.  Design of Explicitly or Implicitly Parallel Low-resolution Character Recognition Algorithms by Means of Genetic Programming , 2002 .

[104]  D. B. Fogel,et al.  Evolving neural networks , 1990, Biological Cybernetics.

[105]  Lutz Prechelt,et al.  Automatic early stopping using cross validation: quantifying the criteria , 1998, Neural Networks.

[106]  William B. Langdon,et al.  Application of Genetic Programming to Induction of Linear Classification Trees , 2000, EuroGP.

[107]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[108]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[109]  Jason H. Moore,et al.  Genetic programming neural networks: A powerful bioinformatics tool for human genetics , 2007, Appl. Soft Comput..

[110]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[111]  Robert E. Smith,et al.  Combined biological paradigms: A neural, genetics-based autonomous systems strategy , 1997, Robotics Auton. Syst..

[112]  E. Cantu-Paz,et al.  An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[113]  Kristina Davoian,et al.  A New Self-Adaptive EP Approach for ANN Weights Training , 2008 .

[114]  Xin Yao,et al.  EPNet for Chaotic Time-Series Prediction , 1996, SEAL.

[115]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[116]  Kumar Chellapilla,et al.  On Making Problems Evolutionarily Friendly - Part 1: Evolving the Most Convenient Representations , 1998, Evolutionary Programming.

[117]  Francisco Sandoval Hernández,et al.  Genetic Synthesis of Discrete-Time Recurrent Neural Network , 1993, IWANN.

[118]  Andries P. Engelbrecht,et al.  A Building Block Approach to Genetic Programming for Rule Discovery , 2002 .