Global optimization algorithms

Gradient descent (GD) is the most popular local optimization algorithm to train multilayer NNs. While GD has shown to be successful in training SUNNs, GD fails to train PUNNs under general assumptions of weight initialization, as shown in the previous chapter. This chapter presents an overview of the following global optimization algorithms: Particle Swarm Optimization (PSO), Genetic Algorithms (GAs) and Leapfrog Optimization (LFOP) . These algorithms are subsequently applied to approximate a set of functions, using PUNNs. The results are compared with that of SUNNs, using gradient descent optimization.

[1]  Lester Ingber,et al.  Adaptive simulated annealing (ASA): Lessons learned , 2000, ArXiv.

[2]  Masaharu Munetomo,et al.  An Adaptive Network Routing Algorithm Employing Path Genetic Operators , 1997, ICGA.

[3]  David J. Weir,et al.  Characterizing mildly context-sensitive grammar formalisms , 1988 .

[4]  Tadahiko Murata,et al.  Genetic network programming with automatically defined groups for assigning proper roles to multiple agents , 2005, GECCO '05.

[5]  Lester Ingber,et al.  Trading markets with canonical momenta and adaptive simulated annealing , 1996 .

[6]  Lester Ingber.  A simple options training model , 1999 .

[7]  Tadahiko Murata,et al.  Developing Cooperation of Multiple Agents Using Genetic Network Programming with Automatically Defined Groups , 2004 .

[8]  Lester Ingber,et al.  Simulated annealing: Practice versus theory , 1993 .

[9]  Masaharu Munetomo,et al.  A migration scheme for the genetic adaptive routing algorithm , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[10]  Kurt Geihs,et al.  Genetic Programming for Proactive Aggregation Protocols , 2007, ICANNGA.

[11]  David E. Goldberg,et al.  Linkage Identification by Non-monotonicity Detection for Overlapping Functions , 1999, Evolutionary Computation.

[12]  Kurt Geihs,et al.  An Adaptable Framework for Distributed Multi-Objective Search Algorithms Applied to the Genetic Programming of Sensor Networks , 2006 .