A tabu search algorithm for the training of neural networks

The most widely used training algorithm of neural networks (NNs) is back propagation (BP), a gradient-based technique that requires significant computational effort. Metaheuristic search techniques such as genetic algorithms, tabu search (TS) and simulated annealing have been recently used to cope with major shortcomings of BP such as the tendency to converge to a local optimal and a slow convergence rate. In this paper, an efficient TS algorithm employing different strategies to provide a balance between intensification and diversification is proposed for the training of NNs. The proposed algorithm is compared with other metaheuristic techniques found in literature using published test problems, and found to outperform them in the majority of the test cases.

[1]  Yutaka Fukuoka,et al.  A modified back-propagation method to avoid false local minima , 1998, Neural Networks.

[2]  Randall S. Sexton,et al.  Toward global optimization of neural networks: A comparison of the genetic algorithm and backpropagation , 1998, Decis. Support Syst..

[3]  D. B. Fogel,et al.  Evolving neural networks , 1990, Biological Cybernetics.

[4]  Duc Truong Pham,et al.  Training Elman and Jordan networks for system identification using genetic algorithms , 1999, Artif. Intell. Eng..

[5]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[6]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[7]  Fred Glover,et al.  Tabu Search - Part II , 1989, INFORMS J. Comput..

[8]  Jatinder N. D. Gupta,et al.  Comparative evaluation of genetic algorithm and backpropagation for training neural networks , 2000, Inf. Sci..

[9]  Tamás D. Gedeon,et al.  Simulated annealing and weight decay in adaptive learning: the SARPROP algorithm , 1998, IEEE Trans. Neural Networks.

[10]  Don R. Hush,et al.  Training a Sigmoidal Node Is Hard , 1999, Neural Computation.

[11]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[12]  Randall S. Sexton,et al.  Comparing backpropagation with a genetic algorithm for neural network training , 1999 .

[13]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[14]  Martin Fodslette Meiller.  A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning , 1993 .

[15]  Richard W. Eglese,et al.  Simulated annealing: A tool for operational research , 1990 .

[16]  James P. Kelly,et al.  A scatter-search-based learning algorithm for neural network training , 1996, J. Heuristics.

[17]  Tom Tollenaere,et al.  SuperSAB: Fast adaptive back propagation with good scaling properties , 1990, Neural Networks.

[18]  BattitiRoberto.  First- and second-order methods for learning , 1992 .

[19]  Fred W. Glover,et al.  Future paths for integer programming and links to artificial intelligence , 1986, Comput. Oper. Res..

[20]  Randall S. Sexton,et al.  Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing , 1999, Eur. J. Oper. Res..

[21]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[22]  Thomas P. Vogl,et al.  Rescaling of variables in back propagation learning , 1991, Neural Networks.

[23]  Etienne Barnard,et al.  Optimization for training neural nets , 1992, IEEE Trans. Neural Networks.

[24]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[25]  T. Warren Liao,et al.  Manufacturing Process Modeling and Optimization Based on Multi-Layer Perceptron Network , 1998 .

[26]  Sung-Bae Cho,et al.  Evolutionary Learning of Modular Neural Networks with Genetic Programming , 1998, Applied Intelligence.

[27]  Roberto Battiti,et al.  Training neural nets with the reactive tabu search , 1995, IEEE Trans. Neural Networks.

[28]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[29]  Isidro Moctezuma-Cantorán,et al.  Manufacturing Process Modeling , 2007 .

[30]  Bahram Alidaee,et al.  Global optimization for artificial neural networks: A tabu search application , 1998, Eur. J. Oper. Res..