Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing

The escalation of Neural Network research in Business has been brought about by the ability of neural networks, as a tool, to closely approximate unknown functions to any degree of desired accuracy. Although, gradient based search techniques such as back-propagation are currently the most widely used optimization techniques for training neural networks, it has been shown that these gradient techniques are severely limited in their ability to find global solutions. Global search techniques have been identified as a potential solution to this problem. In this paper we examine two well known global search techniques, Simulated Annealing and the Genetic Algorithm, and compare their performance. A Monte Carlo study was conducted in order to test the appropriateness of these global search techniques for optimizing neural networks.

[1]  Geraint Johnes,et al.  Up Around the Bend: Linear and nonlinear models of the UK economy compared , 2000 .

[2]  Sandro Ridella,et al.  Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithmCorrigenda for this article is available here , 1987, TOMS.

[3]  Gregory R. Madey,et al.  The Application of Neural Networks and a Qualitative Response Model to the Auditor's Going Concern Uncertainty Decision* , 1995 .

[4]  Robert E. Dorsey,et al.  Genetic algorithms for estimation problems with multiple optima , 1995 .

[5]  Venkat Subramanian,et al.  A GRG2-Based System for Training Neural Networks: Design and Computational Experience , 1993, INFORMS J. Comput..

[6]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[7]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[8]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[9]  Randall S. Sexton,et al.  Toward global optimization of neural networks: A comparison of the genetic algorithm and backpropagation , 1998, Decis. Support Syst..

[10]  Olvi L. Mangasarian,et al.  Mathematical Programming in Neural Networks , 1993, INFORMS J. Comput..

[11]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[12]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[13]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[14]  William Nick Street,et al.  Breast Cancer Diagnosis and Prognosis Via Linear Programming , 1995, Oper. Res..

[15]  William L. Goffe,et al.  SIMANN: FORTRAN module to perform Global Optimization of Statistical Functions with Simulated Annealing , 1992 .

[16]  Manoj Kumar.  OPTIMIZATION USING GENETIC ALGORITHMS , 1998 .

[17]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[18]  Yann LeCun,et al.  Learning processes in an asymmetric threshold network , 1986 .

[19]  D. M. Himmelblau,et al.  Should backpropagation be replaced by more effective optimization algorithms? , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[20]  Bahram Alidaee,et al.  Global optimization for artificial neural networks: A tabu search application , 1998, Eur. J. Oper. Res..

[21]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[22]  Kurt Hornik,et al.  FEED FORWARD NETWORKS ARE UNIVERSAL APPROXIMATORS , 1989 .

[23]  Shouhong Wang,et al.  Application of the Back Propagation Neural Network Algorithm with Monotonicity Constraints for Two‐Group Classification Problems* , 1993 .

[24]  Halbert White,et al.  On learning the derivatives of an unknown mapping with multilayer feedforward networks , 1992, Neural Networks.

[25]  Shouhong Wang.  The unpredictability of standard back propagation neural networks in classification applications , 1995 .

[26]  E. Mine Cinar,et al.  Neural Networks: A New Tool for Predicting Thrift Failures , 1992 .

[27]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .