Global optimization for artificial neural networks: A tabu search application

The ability of neural networks to closely approximate unknown functions to any degree of desired accuracy has generated considerable demand for neural network research in business. The attractiveness of neural network research stems from researchers' need to approximate models within the business environment without having a priori knowledge about the true underlying function. Gradient techniques, such as backpropagation, are currently the most widely used methods for neural network optimization. Since these techniques search for local solutions, they are subject to local convergence and thus can perform poorly even on simple problems when forecasting out-of-sample. Consequently, a global search algorithm is warranted. In this paper we examine tabu search (TS) as a possible alternative to the problematic backpropagation approach. A Monte Carlo study was conducted to test the appropriateness of TS as a global search technique for optimizing neural networks. Holding the neural network architecture constant, 530 independent runs were conducted for each of seven test functions, including a production function that exhibits both increasing and diminishing marginal returns and the Mackey-Glass chaotic time series. In the resulting comparison, TS derived solutions that were significantly superior to those of backpropagation solutions for in-sample, interpolation, and extrapolation test data for all seven test functions. It was also shown that fewer function evaluations were needed to find these optimal values.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[3]  Jianqiang Yi,et al.  Backpropagation based on the logarithmic error function and elimination of local minima , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[4]  Fred W. Glover,et al.  Future paths for integer programming and links to artificial intelligence , 1986, Comput. Oper. Res..

[5]  F. Glover,et al.  The application of tabu search to the symmetric traveling salesman problem , 1989 .

[6]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[7]  D. M. Himmelblau,et al.  Should backpropagation be replaced by more effective optimization algorithms? , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[8]  Jadranka Skorin-Kapov,et al.  Tabu Search Applied to the Quadratic Assignment Problem , 1990, INFORMS J. Comput..

[9]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[10]  William L. Goffe,et al.  SIMANN: FORTRAN module to perform Global Optimization of Statistical Functions with Simulated Annealing , 1992 .

[11]  M.J.J. Holt,et al.  Convergence of back-propagation in neural networks using a log-likelihood cost function , 1990 .

[12]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[13]  Gregory R. Madey,et al.  The Application of Neural Networks and a Qualitative Response Model to the Auditor's Going Concern Uncertainty Decision* , 1995 .

[14]  J. A. Bland,et al.  Tabu search and design optimization , 1991, Comput. Aided Des..

[15]  Randy L. Shimabukuro,et al.  Back propagation learning with trinary quantization of weight updates , 1991, Neural Networks.

[16]  Venkat Subramanian,et al.  A GRG2-Based System for Training Neural Networks: Design and Computational Experience , 1993, INFORMS J. Comput..

[17]  Jihong Lee,et al.  Improvement on function approximation capability of backpropagation neural networks , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[18]  Esther Levin,et al.  Accelerated Learning in Layered Neural Networks , 1988, Complex Syst..

[19]  Josef Hadar,et al.  Mathematical Theory of Economic Behavior , 1971 .

[20]  Kurt Hornik,et al.  FEED FORWARD NETWORKS ARE UNIVERSAL APPROXIMATORS , 1989 .

[21]  Alex Pentland,et al.  Analysis of Neural Networks with Redundancy , 1990, Neural Computation.

[22]  Shouhong Wang,et al.  Application of the Back Propagation Neural Network Algorithm with Monotonicity Constraints for Two‐Group Classification Problems* , 1993 .

[23]  Michel Gendreau,et al.  A tabu search procedure for multicommodity location/allocation with balancing requirements , 1992, Ann. Oper. Res..

[24]  Arjen van Ooyen,et al.  Improving the convergence of the back-propagation algorithm , 1992, Neural Networks.

[25]  J. P. Kelly,et al.  Tabu search for the multilevel generalized assignment problem , 1995 .

[26]  James P. Kelly,et al.  Large-scale controlled rounding using tabu search with strategic oscillation , 1993, Ann. Oper. Res..

[27]  Benjamin W. Wah,et al.  Global Optimization for Neural Network Training , 1996, Computer.

[28]  J. A. Bland.  Nonlinear optimization of constrained functions using tabu search , 1993 .

[29]  Fred Glover,et al.  Tabu Search: A Tutorial , 1990 .

[30]  David E. Rumelhart,et al.  BACK-PROPAGATION, WEIGHT-ELIMINATION AND TIME SERIES PREDICTION , 1991 .

[31]  Raymond L. Watrous.  Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization , 1988 .

[32]  Françoise Fogelman-Soulié,et al.  Disordered Systems and Biological Organization , 1986, NATO ASI Series.

[33]  Jocelyn Sietsma,et al.  Creating artificial neural networks that generalize , 1991, Neural Networks.

[34]  D. Werra,et al.  Tabu search: a tutorial and an application to neural networks , 1989 .

[35]  Geoffrey E. Hinton,et al.  Experiments on Learning by Back Propagation. , 1986 .

[36]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[37]  Yann LeCun,et al.  Learning processes in an asymmetric threshold network , 1986 .

[38]  Shouhong Wang.  The unpredictability of standard back propagation neural networks in classification applications , 1995 .

[39]  E. Mine Cinar,et al.  Neural Networks: A New Tool for Predicting Thrift Failures , 1992 .

[40]  J. A. Bland.  A derivative-free exploratory tool for function minimisation based on tabu search , 1994 .

[41]  Halbert White,et al.  On learning the derivatives of an unknown mapping with multilayer feedforward networks , 1992, Neural Networks.

[42]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[43]  F. Glover,et al.  Bandwidth packing: a tabu search approach , 1993 .