Neural network methods in combinatorial optimization

Abstract We describe two basic approaches to using neural networks for optimization. The more popular approach is to formulate a combinatorial optimization task in terms of minimizing a cost function. Neural network models have been developed or interpreted as minimization machines. Before using a network to solve a problem, one must express the problem as a mathematical function that is to be minimized. The other basic approach is to design competition-based neural networks in which neurons are allowed to compete to become active under certain conditions. These approaches suggest neural network methods as an alternative for solving certain optimization tasks as compared to classical optimization techniques and other novel approaches like simulated annealing. Theoretical results on the power of neural networks for solving difficult problems will be reviewed. We provide a list of optimization problems which have been tested on neural networks. In particular, we take a closer look at the neural network methods for solving the traveling salesman problem and provide a categorization of the solution methods. We also discuss the application of neural networks to constraint satisfaction problems. A comprehensive bibliography is provided to facilitate further investigation for the interested reader.

[1]  E. Bonomi,et al.  The N-City Travelling Salesman Problem: Statistical Mechanics and the Metropolis Algorithm , 1984 .

[2]  Wesley E. Snyder,et al.  Optimization by Mean Field Annealing , 1988, NIPS.

[3]  Chinchuan Chiu,et al.  An Artificial Neural Network Algorithm for Dynamic Programming , 1990, Int. J. Neural Syst..

[4]  W.S. Wong,et al.  An elastic net solution to obstacle avoidance tour planning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[5]  Leon O. Chua,et al.  Neural networks for nonlinear programming , 1988 .

[6]  Mark Derthick,et al.  'Ensemble' Boltzmann Units have Collective Computational Properties like those of Hopfield and Tank Neurons , 1987, NIPS.

[7]  Carsten Peterson,et al.  A New Method for Mapping Optimization Problems Onto Neural Networks , 1989, Int. J. Neural Syst..

[8]  J. A. Hertz,et al.  A network system for image segmentation , 1989, International 1989 Joint Conference on Neural Networks.

[9]  David E. van den Bout,et al.  A traveling salesman objective function that works , 1988, IEEE 1988 International Conference on Neural Networks.

[10]  B. K. Jenkins,et al.  Image restoration using a neural network , 1988, IEEE Trans. Acoust. Speech Signal Process..

[11]  Hahn-Ming Lee,et al.  Neural network processing through energy minimization with learning ability to the multiconstraint zero-one knapsack problem , 1989, [Proceedings 1989] IEEE International Workshop on Tools for Artificial Intelligence.

[12]  Mark D. Johnston,et al.  A discrete stochastic neural network algorithm for constraint satisfaction problems , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[13]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[14]  A. Moopenn,et al.  A neural network for Euclidean distance minimization , 1988, IEEE 1988 International Conference on Neural Networks.

[15]  C Koch,et al.  Analog "neuronal" networks in early vision. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[16]  Rama Chellappa,et al.  Artificial neural network algorithms for some computer vision problems , 1988 .

[17]  Mahesan Niranjan,et al.  A theoretical investigation into the performance of the Hopfield model , 1990, IEEE Trans. Neural Networks.

[18]  T. Poggio,et al.  A parallel algorithm for real-time computation of optical flow , 1989, Nature.

[19]  Rama Chellappa,et al.  Stereo matching using a neural network , 1988, ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing.

[20]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[21]  X. Xu,et al.  Effective neural algorithms for the traveling salesman problem , 1991, Neural Networks.

[22]  Li-Chen Fu,et al.  Ineffectiveness in solving combinatorial optimization problems using a Hopfield network: a new perspective from aliasing effect , 1990, IJCNN.

[23]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[24]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Joseph W. Goodman,et al.  On the power of neural networks for solving hard problems , 1990, J. Complex..

[26]  Robert M. Haralick,et al.  Structural Descriptions and Inexact Matching , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[28]  P. Sadayappan,et al.  Optimization by neural networks , 1988, IEEE 1988 International Conference on Neural Networks.

[29]  Rina Dechter,et al.  Constraint Directed Reasoning , 1988 .

[30]  V. C. Barbosa,et al.  Feasible directions linear programming by neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[31]  Carsten Peterson,et al.  Neural Networks and NP-complete Optimization Problems; A Performance Study on the Graph Bisection Problem , 1988, Complex Syst..

[32]  Behrooz Kamgar-Parsi,et al.  On problem solving with Hopfield neural networks , 1990, International 1989 Joint Conference on Neural Networks.

[33]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[34]  N. Toomarian,et al.  A comparison study of optimization methods for the bipartite matching problem (BMP) , 1988, IEEE 1988 International Conference on Neural Networks.

[35]  Simon Kasif,et al.  On the Parallel Complexity of Discrete Relaxation in Constraint Satisfaction Networks , 1990, Artif. Intell..

[36]  Dejan J. Sobajic,et al.  Neural Networks and Knowledge Engineering , 1991, IEEE Trans. Knowl. Data Eng..

[37]  Tao Li,et al.  Design of Competition-Based Neural Networks for Combinatorial Optimization , 1990, Int. J. Neural Syst..

[38]  Laurence A. Wolsey,et al.  Integer and Combinatorial Optimization , 1988 .

[39]  Gintaras V. Reklaitis,et al.  Generalized Hopfield Networks and Nonlinear Optimization , 1989, NIPS.

[40]  Michael J. Daily,et al.  Color image segmentation using Markov random fields , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[41]  Sweet.  Determination of parameters in a Hopfield/Tank computational network , 1988 .

[42]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[43]  Carsten Peterson,et al.  "Teachers and Classes" with Neural Networks , 1991, Int. J. Neural Syst..

[44]  Alan K. Mackworth.  Constraint Satisfaction , 1985 .

[45]  Carsten Peterson,et al.  Parallel Distributed Approaches to Combinatorial Optimization: Benchmark Studies on Traveling Salesman Problem , 1990, Neural Computation.

[46]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[47]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[48]  L. B. Moorehead,et al.  A neural network for shape recognition , 1988, IEEE Region 5 Conference, 1988: 'Spanning the Peaks of Electrotechnology'.

[49]  Steven W. Zucker,et al.  On the Foundations of Relaxation Labeling Processes , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  Yoshiyasu Takefuji,et al.  Integer linear programming neural networks for job-shop scheduling , 1988, IEEE 1988 International Conference on Neural Networks.

[51]  Eric Goles,et al.  Lyapunov function for parallel neural networks , 1987 .

[52]  Bernard Angéniol,et al.  Self-organizing feature maps and the travelling salesman problem , 1988, Neural Networks.

[53]  R. Rosner,et al.  Optimization algorithms: simulated annealing and neural network processing , 1986 .

[54]  C. Koch,et al.  Computing motion in the presence of discontinuities: algorithm and analog networks , 1988 .

[55]  Yuichiro Anzai,et al.  Solving large scale puzzles with neural networks , 1989, [Proceedings 1989] IEEE International Workshop on Tools for Artificial Intelligence.

[56]  John C. Platt,et al.  Constrained Differential Optimization , 1987, NIPS.

[57]  David E. van den Bout,et al.  Graph partitioning using annealed neural networks , 1990, International 1989 Joint Conference on Neural Networks.

[58]  H. Szu,et al.  Fast TSP algorithm based on binary neuron output and analog neuron input using the zero-diagonal interconnect matrix and necessary and sufficient constraints of the permutation matrix , 1988, IEEE 1988 International Conference on Neural Networks.

[59]  Alan L. Yuille,et al.  Generalized Deformable Models, Statistical Physics, and Matching Problems , 1990, Neural Computation.

[60]  Norberto M. Grzywacz,et al.  A computational theory for the perception of coherent visual motion , 1988, Nature.

[61]  Heinz Mühlenbein,et al.  Evolution algorithms in combinatorial optimization , 1988, Parallel Comput..

[62]  Joseph W. Goodman,et al.  A generalized convergence theorem for neural networks , 1988, IEEE Trans. Inf. Theory.

[63]  Charles L. Britton,et al.  Neural network models for Linear Programming , 1989 .

[64]  Petar D. Simic,et al.  Statistical mechanics as the underlying theory of ‘elastic’ and ‘neural’ optimisations , 1990 .

[65]  Vladimir Cherkassky,et al.  Scaling neural network for job-shop scheduling , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[66]  David E. van den Bout,et al.  Encoding logical constraints into neural network cost functions , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[67]  Richard Szeliski,et al.  An Analysis of the Elastic Net Approach to the Traveling Salesman Problem , 1989, Neural Computation.

[68]  R. C. Mann,et al.  An extension of the Hopfield-Tank model for solution of the multiple traveling salesmen problem , 1988, IEEE 1988 International Conference on Neural Networks.

[69]  Gene Gindi,et al.  OPTICAL NEUROCOMPUTER FOR IMPLEMENTATION OF THE MARR-POGGIO STEREO ALGORITHM. , 1987 .

[70]  P. Bozovsky.  Discrete Hopfield model with graded response (analysis and applications) , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[71]  Nasser M. Nasrabadi,et al.  Integration of stereo vision and optical flow using Markov random fields , 1988, ICNN.