Evolutionary Computation and Its Applications in Neural and Fuzzy Systems

Neural networks and fuzzy systems are two soft-computing paradigms for system modelling. Adapting a neural or fuzzy system requires to solve two optimization problems: structural optimization and parametric optimization. Structural optimization is a discrete optimization problem which is very hard to solve using conventional optimization techniques. Parametric optimization can be solved using conventional optimization techniques, but the solution may be easily trapped at a bad local optimum. Evolutionary computation is a general-purpose stochastic global optimization approach under the universally accepted neo-Darwinian paradigm, which is a combination of the classical Darwinian evolutionary theory, the selectionism of Weismann, and the genetics of Mendel. Evolutionary algorithms are a major approach to adaptation and optimization. In this paper, we first introduce evolutionary algorithms with emphasis on genetic algorithms and evolutionary strategies. Other evolutionary algorithms such as genetic programming, evolutionary programming, particle swarm optimization, immune algorithm, and ant colony optimization are also described. Some topics pertaining to evolutionary algorithms are also discussed, and a comparison between evolutionary algorithms and simulated annealing is made. Finally, the application of EAs to the learning of neural networks as well as to the structural and parametric adaptations of fuzzy systems is also detailed.

[1]  Nikhil,et al.  Directed Mutation in Genetic Algorithms , 2022 .

[2]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[3]  Chin-Teng Lin,et al.  A GA-based fuzzy adaptive learning control network , 2000, Fuzzy Sets Syst..

[4]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[5]  J. AllenF,et al.  The American Naturalist Vol , 1897 .

[6]  Shigeyoshi Tsutsui,et al.  Forking Genetic Algorithms: GAs with Search Space Division Schemes , 1997, Evolutionary Computation.

[7]  James C. Bean,et al.  Genetic Algorithms and Random Keys for Sequencing and Optimization , 1994, INFORMS J. Comput..

[8]  Alberto Alvarez,et al.  A Neural Network with Evolutionary Neurons , 2002, Neural Processing Letters.

[9]  Francisco Herrera,et al.  Dynamic and heuristic fuzzy connectives-based crossover operators for controlling the diversity and convergence of real-coded genetic algorithms , 1996, Int. J. Intell. Syst..

[10]  Maurice Clerc,et al.  The particle swarm - explosion, stability, and convergence in a multidimensional complex space , 2002, IEEE Trans. Evol. Comput..

[11]  Luca Maria Gambardella,et al.  Ant Algorithms for Discrete Optimization , 1999, Artificial Life.

[12]  John J. Grefenstette,et al.  Deception Considered Harmful , 1992, FOGA.

[13]  L. Darrell Whitley,et al.  Modeling Simple Genetic Algorithms for Permutation Problems , 1994, FOGA.

[14]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[15]  Lawrence Davis,et al.  Bit-Climbing, Representational Bias, and Test Suite Design , 1991, ICGA.

[16]  Martin J. Oates,et al.  The Pareto Envelope-Based Selection Algorithm for Multi-objective Optimisation , 2000, PPSN.

[17]  R. E. Uhrig,et al.  Using genetic algorithms to select inputs for neural networks , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[18]  Marco Laumanns,et al.  SPEA2: Improving the strength pareto evolutionary algorithm , 2001 .

[19]  Thomas Bäck,et al.  An Overview of Evolutionary Algorithms for Parameter Optimization , 1993, Evolutionary Computation.

[20]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[21]  Ian C. Parmee,et al.  The Ant Colony Metaphor for Searching Continuous Design Spaces , 1995, Evolutionary Computing, AISB Workshop.

[22]  Jerne Nk.  Towards a network theory of the immune system. , 1974 .

[23]  Bernard Manderick,et al.  Fine-Grained Parallel Genetic Algorithms , 1989, ICGA.

[24]  Jürgen Schmidhuber,et al.  Self-organizing nets for optimization , 2004, IEEE Transactions on Neural Networks.

[25]  M. Dorigo,et al.  1 Positive Feedback as a Search Strategy , 1991 .

[26]  Hugh M. Cartwright,et al.  The Application of the Genetic Algorithm to Two-Dimensional Strings: The Source Apportionment Problem , 1993, ICGA.

[27]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[28]  John R. Koza,et al.  Genetic programming (videotape): the movie , 1992 .

[29]  Kim W. C. Ku,et al.  Approaches to combining local and evolutionary search for training neural networks: a review and some new results , 2003 .

[30]  Fernando José Von Zuben,et al.  Learning and optimization using the clonal selection principle , 2002, IEEE Trans. Evol. Comput..

[31]  Licheng Jiao,et al.  A novel genetic algorithm based on immunity , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[32]  Hao Chen,et al.  Parallel Genetic Simulated Annealing: A Massively Parallel SIMD Algorithm , 1998, IEEE Trans. Parallel Distributed Syst..

[33]  George W. Irwin,et al.  Fuzzy coding of genetic algorithms , 2003, IEEE Trans. Evol. Comput..

[34]  Robert J. Marks,et al.  Dynamic fuzzy control of genetic algorithm parameter coding , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[35]  Christian W. Dawson,et al.  A review of genetic algorithms applied to training radial basis function networks , 2004, Neural Computing & Applications.

[36]  X. Yao,et al.  Fast evolutionary algorithms , 2003 .

[37]  Gheorghe Paun,et al.  Membrane Computing , 2002, Natural Computing Series.

[38]  Marco Russo,et al.  FuGeNeSys-a fuzzy genetic neural system for fuzzy modeling , 1998, IEEE Trans. Fuzzy Syst..

[39]  Filippo Menczer,et al.  Local Selection , 1998, Evolutionary Programming.

[40]  Heinz Mühlenbein,et al.  Parallel Genetic Algorithms, Population Genetics, and Combinatorial Optimization , 1989, Parallelism, Learning, Evolution.

[41]  Michael D. Vose,et al.  Modeling genetic algorithms with Markov chains , 1992, Annals of Mathematics and Artificial Intelligence.

[42]  Hyeonjoong Cho,et al.  Population-oriented simulated annealing technique based on local temperature concept , 1998 .

[43]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[44]  Kalyanmoy Deb,et al.  RapidAccurate Optimization of Difficult Problems Using Fast Messy Genetic Algorithms , 1993, ICGA.

[45]  L. Darrell Whitley,et al.  Adding Learning to the Cellular Development of Neural Networks: Evolution and the Baldwin Effect , 1993, Evolutionary Computation.

[46]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[47]  Witold Pedrycz,et al.  Fuzzy evolutionary computing , 1998, Soft Comput..

[48]  David I. Lewin,et al.  DNA computing , 2002, Comput. Sci. Eng..

[49]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[50]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[51]  Stephanie Forrest,et al.  Analogies with immunology represent an important step toward the vision of robust, distributed protection for computers. , 1991 .

[52]  Gilbert Syswerda,et al.  Uniform Crossover in Genetic Algorithms , 1989, ICGA.

[53]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[54]  LarrañagaP.,et al.  Genetic Algorithms for the Travelling Salesman Problem , 1999 .

[55]  Barbara Webb,et al.  Swarm Intelligence: From Natural to Artificial Systems , 2002, Connect. Sci..

[56]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[57]  Adam Prügel-Bennett,et al.  Symmetry breaking in population-based optimization , 2004, IEEE Transactions on Evolutionary Computation.

[58]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[59]  Kalyanmoy Deb,et al.  Messy Genetic Algorithms: Motivation, Analysis, and First Results , 1989, Complex Syst..

[60]  Rajarshi Das,et al.  A Study of Control Parameters Affecting Online Performance of Genetic Algorithms for Function Optimization , 1989, ICGA.

[61]  David B. Fogel,et al.  Evolutionary Computation: The Fossil Record , 1998 .

[62]  R. Lewontin.  ‘The Selfish Gene’ , 1977, Nature.

[63]  Seong-Whan Lee.  Off-Line Recognition of Totally Unconstrained Handwritten Numerals Using Multilayer Cluster Neural Network , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[65]  Carlos A. Coello Coello,et al.  Handling multiple objectives with particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.

[66]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[67]  R. Reynolds.  AN INTRODUCTION TO CULTURAL ALGORITHMS , 2008 .

[68]  Filippo Menczer,et al.  Efficient and Scalable Pareto Optimization by Evolutionary Local Selection Algorithms , 2000, Evolutionary Computation.

[69]  Marco Russo,et al.  Genetic fuzzy learning , 2000, IEEE Trans. Evol. Comput..

[70]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[71]  L. Darrell Whitley,et al.  Scheduling Problems and Traveling Salesmen: The Genetic Edge Recombination Operator , 1989, International Conference on Genetic Algorithms.

[72]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[73]  Alden H. Wright,et al.  Genetic Algorithms for Real Parameter Optimization , 1990, FOGA.

[74]  Uday Kumar Chakraborty,et al.  An analysis of Gray versus binary encoding in genetic search , 2003, Inf. Sci..

[75]  Sina Balkir,et al.  Evolution-based design of neural fuzzy networks using self-adapting genetic parameters , 2002, IEEE Trans. Fuzzy Syst..

[76]  Juan Julián Merelo Guervós,et al.  Evolving Multilayer Perceptrons , 2000, Neural Processing Letters.

[77]  Gunar E. Liepins,et al.  Punctuated Equilibria in Genetic Search , 1991, Complex Syst..

[78]  Yoh-Han Pao,et al.  Combinatorial optimization with use of guided evolutionary simulated annealing , 1995, IEEE Trans. Neural Networks.

[79]  N K Jerne,et al.  Towards a network theory of the immune system. , 1973, Annales d'immunologie.

[80]  L. Darrell Whitley,et al.  Lamarckian Evolution, The Baldwin Effect and Function Optimization , 1994, PPSN.

[81]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[82]  L. Darrell Whitley,et al.  Fundamental Principles of Deception in Genetic Search , 1990, FOGA.

[83]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[84]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[85]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[86]  Terence C. Fogarty,et al.  Varying the Probability of Mutation in the Genetic Algorithm , 1989, ICGA.

[87]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[88]  Manuel López-Ibáñez,et al.  Ant colony optimization , 2010, GECCO '10.

[89]  Pablo Moscato,et al.  Memetic algorithms: a short introduction , 1999 .

[90]  Pablo Moscato,et al.  On Evolution, Search, Optimization, Genetic Algorithms and Martial Arts : Towards Memetic Algorithms , 1989 .

[91]  G L Ada,et al.  The clonal-selection theory. , 1987, Scientific American.

[92]  John J. Grefenstette,et al.  Genetic Algorithms for the Traveling Salesman Problem , 1985, ICGA.

[93]  Kennetb A. De.  Genetic Algorithms Are NOT Function Optimizers , 1992 .

[94]  Witold Pedrycz,et al.  Evolutionary fuzzy modeling , 2003, IEEE Trans. Fuzzy Syst..

[95]  L. Darrell Whitley,et al.  GENITOR II: a distributed genetic algorithm , 1990, J. Exp. Theor. Artif. Intell..

[96]  David W. Corne,et al.  Approximating the Nondominated Front Using the Pareto Archived Evolution Strategy , 2000, Evolutionary Computation.

[97]  Y. Pao,et al.  Automatic Optimal Design of Fuzzy Systems Based on Universal Approximation and Evolutionary Programming , 1995 .

[98]  Thomas Martinetz,et al.  'Neural-gas' network for vector quantization and its application to time-series prediction , 1993, IEEE Trans. Neural Networks.

[99]  Zbigniew Michalewicz,et al.  A Survey of Constraint Handling Techniques in Evolutionary Computation Methods , 1995 .

[100]  Yue Shi,et al.  A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[101]  Hideyuki Takagi,et al.  Dynamic Control of Genetic Algorithms Using Fuzzy Logic Techniques , 1993, ICGA.

[102]  M.N.S. Swamy,et al.  Neural networks in a softcomputing framework , 2006 .

[103]  Jonathan Baxter.  The evolution of learning algorithms for artificial neural networks , 1993 .

[104]  Daniel Raymond Frantz,et al.  Nonlinearities in genetic adaptive search. , 1972 .

[105]  Günter Rudolph,et al.  Convergence analysis of canonical genetic algorithms , 1994, IEEE Trans. Neural Networks.

[106]  Yuval Davidor,et al.  A Naturally Occurring Niche and Species Phenomenon: The Model and First Results , 1991, ICGA.

[107]  Ralph R. Martin,et al.  A Sequential Niche Technique for Multimodal Function Optimization , 1993, Evolutionary Computation.

[108]  David E. Goldberg,et al.  A niched Pareto genetic algorithm for multiobjective optimization , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[109]  V. Delport.  Codebook design in vector quantisation using a hybrid system of parallel simulated annealing and evolutionary selection , 1996 .

[110]  Yuji Sato,et al.  2-D Genetic Algorithms for Determining Neural Network Structure and Weights , 1995, Evolutionary Programming.

[111]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[112]  Michael Herdy,et al.  Application of the 'Evolutionsstrategie' to Discrete Optimization Problems , 1990, PPSN.

[113]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[114]  Sung-Kwun Oh,et al.  Multi-layer hybrid fuzzy polynomial neural networks: a design in the framework of computational intelligence , 2005, Neurocomputing.

[115]  Reinhard Männer,et al.  Towards an Optimal Mutation Probability for Genetic Algorithms , 1990, PPSN.

[116]  Francisco Herrera,et al.  Fuzzy adaptive genetic algorithms: design, taxonomy, and future directions , 2003, Soft Comput..

[117]  David R. Jefferson,et al.  Selection in Massively Parallel Genetic Algorithms , 1991, ICGA.

[118]  Nikolaus Hansen,et al.  Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[119]  Zbigniew Michalewicz,et al.  GAVaPS-a genetic algorithm with varying population size , 1994, Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence.

[120]  David E. Goldberg,et al.  OMEGA - Ordering Messy GA: Solving Permutation Problems with the Fast Genetic Algorithm and Random Keys , 2000, GECCO.

[121]  Heinz Mühlenbein,et al.  Analysis of Selection, Mutation and Recombination in Genetic Algorithms , 1995, Evolution and Biocomputation.

[122]  Larry J. Eshelman,et al.  The CHC Adaptive Search Algorithm: How to Have Safe Search When Engaging in Nontraditional Genetic Recombination , 1990, FOGA.

[123]  Peter D. Turney.  Myths and Legends of the Baldwin Effect , 2002, ICML 2002.

[124]  Witold Pedrycz,et al.  Fuzzy-set based models of neurons and knowledge-based networks , 1993, IEEE Trans. Fuzzy Syst..

[125]  Filippo Menczer,et al.  Evidence of hyperplanes in the genetic learning of neural networks , 2004, Biological Cybernetics.

[126]  D. Wolpert,et al.  No Free Lunch Theorems for Search , 1995 .

[127]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[128]  B. R. Fox,et al.  Genetic Operators for Sequencing Problems , 1990, FOGA.

[129]  Chang-Yong Lee,et al.  Entropy-Boltzmann selection in the genetic algorithms , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[130]  Odej Kao,et al.  The Effects of Partial Restarts in Evolutionary Search , 2001, Artificial Evolution.

[131]  Kyu Ho Park,et al.  Fast learning method for back-propagation neural network by evolutionary adaptation of learning rates , 1996, Neurocomputing.

[132]  L. Darrell Whitley,et al.  Changing Representations During Search: A Comparative Study of Delta Coding , 1994, Evolutionary Computation.

[133]  I. Harvey,et al.  On recombination and optimal mutation rates , 1999 .

[134]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[135]  Alex A. Freitas,et al.  Evolutionary Computation , 2002 .

[136]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[137]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[138]  Mika Hirvensalo,et al.  Introduction to Evolutionary Computing , 2002, Natural Computing Series.

[139]  Juan Julián Merelo Guervós,et al.  G-Prop: Global optimization of multilayer perceptrons using GAs , 2000, Neurocomputing.

[140]  Jim Smith,et al.  Replacement Strategies in Steady State Genetic Algorithms: Static Environments , 1998, FOGA.

[141]  Luca Maria Gambardella,et al.  Ant colony system: a cooperative learning approach to the traveling salesman problem , 1997, IEEE Trans. Evol. Comput..

[142]  Kalyanmoy Deb,et al.  Muiltiobjective Optimization Using Nondominated Sorting in Genetic Algorithms , 1994, Evolutionary Computation.

[143]  H.-J. Zimmermann,et al.  Intelligent system design support by fuzzy-multi-criteria decision making and/or evolutionary algorithms , 1995, Proceedings of 1995 IEEE International Conference on Fuzzy Systems..

[144]  Henri Atlan,et al.  Theories of Immune Networks , 1989, Springer Series in Synergetics.

[145]  J. Baldwin.  A New Factor in Evolution , 1896, The American Naturalist.

[146]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[147]  V. Delport.  Parallel simulated annealing and evolutionary selection for combinatorial optimisation , 1998 .

[148]  J. D. Schaffer,et al.  Some experiments in machine learning using vector evaluated genetic algorithms (artificial intelligence, optimization, adaptation, pattern recognition) , 1984 .