A review of Hopfield neural networks for solving mathematical programming problems

The Hopfield neural network (HNN) is one major neural network (NN) for solving optimization or mathematical programming (MP) problems. The major advantage of HNN is in its structure can be realized on an electronic circuit, possibly on a VLSI (very large-scale integration) circuit, for an on-line solver with a parallel-distributed process. The structure of HNN utilizes three common methods, penalty functions, Lagrange multipliers, and primal and dual methods to construct an energy function. When the function reaches a steady state, an approximate solution of the problem is obtained. Under the classes of these methods, we further organize HNNs by three types of MP problems: linear, non-linear, and mixed-integer. The essentials of each method are also discussed in details. Some remarks for utilizing HNN and difficulties are then addressed for the benefit of successive investigations. Finally, conclusions are drawn and directions for future study are provided.

[1]  Y. Takahashi,et al.  Mathematical improvement of the Hopfield model for feasible solutions to the traveling salesman problem by a synapse dynamical system , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[2]  Edwin K. P. Chong,et al.  An analysis of a class of neural networks for solving linear programming problems , 1999, IEEE Trans. Autom. Control..

[3]  Mengkang Peng,et al.  An Investigation into the Improvement of Local Minima of the Hopfield Network , 1996, Neural Networks.

[4]  Bahram Alidaee,et al.  Global optimization for artificial neural networks: A tabu search application , 1998, Eur. J. Oper. Res..

[5]  Naoki Kurita,et al.  On the Hopfield Neural Networks and Mean Field Theory , 1996, Neural Networks.

[6]  Marimuthu Palaniswami,et al.  Neural techniques for combinatorial optimization with applications , 1998, IEEE Trans. Neural Networks.

[7]  Enrique Mérida Casermeiro,et al.  An Efficient Multivalued Hopfield Network for the Traveling Salesman Problem , 2001, Neural Processing Letters.

[8]  Jehoshua Bruck,et al.  A study on neural networks , 1988, Int. J. Intell. Syst..

[9]  M.N.S. Swamy,et al.  Neural networks in a softcomputing framework , 2006 .

[10]  Masaki Kobayashi,et al.  Cooperative updating in the Hopfield model , 2001, IEEE Trans. Neural Networks.

[11]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[12]  Ángel Rodríguez-Vázquez,et al.  Switched-capacitor neural networks for linear programming , 1988 .

[13]  Jessica Lowell.  Neural Network , 2001 .

[14]  Jun Wang.  Analogue neural network for solving the assignment problem , 1992 .

[15]  Stefen Hui,et al.  Solving linear programming problems with neural networks: a comparative study , 1995, IEEE Trans. Neural Networks.

[16]  Gerhard Reinelt,et al.  TSPLIB - A Traveling Salesman Problem Library , 1991, INFORMS J. Comput..

[17]  Nezam Mahdavi-Amiri,et al.  An efficient simplified neural network for solving linear and quadratic programming problems , 2006, Appl. Math. Comput..

[18]  Michael P. Walsh,et al.  Augmented Hopfield network for mixed-integer programming , 1999, IEEE Trans. Neural Networks.

[19]  George Papageorgiou,et al.  Improved exploration in Hopfield network state-space through parameter perturbation driven by simulated annealing , 1998, Eur. J. Oper. Res..

[20]  Mohamad H. Hassoun,et al.  A coupled gradient network approach for static and temporal mixed-integer optimization , 1996, IEEE Trans. Neural Networks.

[21]  Alaeddin Malek,et al.  Primal-dual solution for the linear programming problems using neural networks , 2005, Appl. Math. Comput..

[22]  Mitsuo Gen,et al.  Lagrangian ANN for convex programming with linear constraints , 1997 .

[23]  Kate Smith-Miles,et al.  Neural Networks for Combinatorial Optimization: A Review of More Than a Decade of Research , 1999, INFORMS J. Comput..

[24]  G. Croes.  A Method for Solving Traveling-Salesman Problems , 1958 .

[25]  T. Kohonen.  Self-organized formation of topographically correct feature maps , 1982 .

[26]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[27]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[28]  Stan Z. Li,et al.  Improving convergence and solution quality of Hopfield-type neural networks with augmented Lagrange multipliers , 1996, IEEE Trans. Neural Networks.

[29]  Ue-Pyng Wen,et al.  A neural network approach to multiobjective and multilevel programming problems , 2004 .

[30]  Kate A. Smith,et al.  Neural Networks for Combinatorial Optimization: a Review of More Than a Decade of Research , 1999 .

[31]  G. Pawley,et al.  On the stability of the Travelling Salesman Problem algorithm of Hopfield and Tank , 2004, Biological Cybernetics.

[32]  Pedro M. Talaván,et al.  Parameter setting of the Hopfield network applied to TSP , 2002, Neural Networks.

[33]  J.J. Hopfield,et al.  Artificial neural networks , 1988, IEEE Circuits and Devices Magazine.

[34]  Marimuthu Palaniswami,et al.  Traditional heuristic versus Hopfield neural network approaches to a car sequencing problem , 1996 .

[35]  Randall S. Sexton,et al.  Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealing , 1999, Eur. J. Oper. Res..

[36]  Jun Wang,et al.  A primal-dual neural network for online resolving constrained kinematic redundancy in robot motion control , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[37]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[38]  Ramesh Sharda,et al.  Neural Networks for the MS/OR Analyst: An Application Bibliography , 1994 .

[39]  Shigeo Abe.  Global convergence and suppression of spurious states of the Hopfield neural networks , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[40]  Kumar,et al.  Neural Networks a Classroom Approach , 2004 .

[41]  Youshen Xia.  A new neural network for solving linear programming problems and its application , 1996, IEEE Trans. Neural Networks.

[42]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[43]  Leon O. Chua,et al.  Neural networks for nonlinear programming , 1988 .

[44]  Abdesselam Bouzerdoum,et al.  Neural network for quadratic optimization with bound constraints , 1993, IEEE Trans. Neural Networks.

[45]  Chee-Kit Looi,et al.  Neural network methods in combinatorial optimization , 1992, Comput. Oper. Res..

[46]  Jun Wang,et al.  A general methodology for designing globally convergent optimization neural networks , 1998, IEEE Trans. Neural Networks.

[47]  Richard W. Prager,et al.  Limitations of neural networks for solving traveling salesman problems , 1995, IEEE Trans. Neural Networks.

[48]  Michael A. Shanblatt,et al.  Linear and quadratic programming neural network analysis , 1992, IEEE Trans. Neural Networks.

[49]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[50]  Behrooz Shirazi,et al.  Critical analysis of applying Hopfield neural net model to optimization problems , 1989, Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics.

[51]  Bernardete Ribeiro,et al.  A neural network for shortest path computation , 2001, IEEE Trans. Neural Networks.

[52]  Xiang-Sun Zhang,et al.  Neural networks in optimization , 2000 .

[53]  S. H. Huang,et al.  Applications of neural networks in manufacturing: a state-of-the-art survey , 1995 .

[54]  Peter Kwong-Shun Tam,et al.  A Neural Network Methodology of Quadratic Optimization , 1999, Int. J. Neural Syst..

[55]  Behrooz Kamgar-Parsi,et al.  On problem solving with Hopfield neural networks , 1990, International 1989 Joint Conference on Neural Networks.

[56]  Sohrab Effati,et al.  A new nonlinear neural network for solving convex nonlinear programming problems , 2005, Appl. Math. Comput..

[57]  Shuzhi Sam Ge,et al.  On parameter settings of Hopfield networks applied to traveling salesman problems , 2005, IEEE Transactions on Circuits and Systems I: Regular Papers.

[58]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[59]  Philip E. Gill,et al.  Practical optimization , 1981 .

[60]  I A Basheer,et al.  Artificial neural networks: fundamentals, computing, design, and application. , 2000, Journal of microbiological methods.

[61]  Wagner Caradori do Amaral,et al.  Design and analysis of an efficient neural network model for solving nonlinear optimization problems , 2005, Int. J. Syst. Sci..

[62]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[63]  Mark J. O'Malley,et al.  A Lagrangian augmented Hopfield network for mixed integer non-linear programming problems , 2002, Neurocomputing.

[64]  Karl Ricanek,et al.  Hopfield like networks for pattern recognition with application to face recognition , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[65]  Jun Wang.  Primal and dual neural networks for shortest-path routing , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[66]  Amparo Ruiz-Sepúlveda,et al.  Improving local minima of Hopfield networks with augmented Lagrange multipliers for large scale TSPs , 2000, Neural Networks.

[67]  Jun Wang.  A recurrent neural network for solving the shortest path problem , 1996 .

[68]  Paul Juell,et al.  Neural Networks for Selective Vehicle Routing Heuristics , 1990, INFORMS J. Comput..

[69]  Hector A. Rosales-Macedo.  Nonlinear Programming: Theory and Algorithms (2nd Edition) , 1993 .

[70]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[71]  S. Abe.  Global convergence and suppression of spurious states of the Hopfield neural networks , 1993 .

[72]  Bernard Widrow,et al.  Neural networks: applications in industry, business and science , 1994, CACM.

[73]  Subramaniam Balakrishnan,et al.  A neural network to enhance local search in the permutation flowshop , 2005, Comput. Ind. Eng..

[74]  Shengwei Zhang,et al.  Lagrange programming neural networks , 1992 .

[75]  Ue-Pyng Wen,et al.  A hybrid neural network approach to bilevel programming problems , 2007, Appl. Math. Lett..

[76]  M. J. Short,et al.  Large-scale economic dispatch using an improved Hopfield neural network , 1997 .

[77]  Youshen Xia,et al.  Neural network for solving linear programming problems with bounded variables , 1995, IEEE Trans. Neural Networks.

[78]  James P. Ignizio,et al.  Neural networks and operations research: An overview , 1992, Comput. Oper. Res..

[79]  Jun Wang.  Analysis and design of a recurrent neural network for linear programming , 1993 .

[80]  Jacek M. Zurada,et al.  Introduction to artificial neural systems , 1992 .

[81]  Wagner Caradori do Amaral,et al.  Neural Approach for Solving Several Types of Optimization Problems , 2006 .

[82]  Bo K. Wong,et al.  A bibliography of neural network business applications research: 1994-1998 , 2000, Comput. Oper. Res..

[83]  Kazuyuki Aihara,et al.  Chaotic simulated annealing by a neural network model with transient chaos , 1995, Neural Networks.

[84]  Zhang Yi,et al.  Improving local minima of columnar competitive model for TSPs , 2006, IEEE Transactions on Circuits and Systems I: Regular Papers.

[85]  Mokhtar S. Bazaraa,et al.  Nonlinear Programming: Theory and Algorithms , 1993 .

[86]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[87]  Yoshikane Takahashi,et al.  A neural network theory for constrained optimization , 1999, Neurocomputing.

[88]  W K Chen,et al.  A high-performance neural network for solving linear and quadratic programming problems , 1996, IEEE Trans. Neural Networks.

[89]  Chih-Chin Lai,et al.  A neural network for linear matrix inequality problems , 2000, IEEE Trans. Neural Networks Learn. Syst..

[90]  Jatinder N. D. Gupta,et al.  Neural networks in business: techniques and applications for the operations researcher , 2000, Comput. Oper. Res..

[91]  Ivica Kostanic,et al.  Principles of Neurocomputing for Science and Engineering , 2000 .

[92]  Martin T. Hagan,et al.  Neural network design , 1995 .

[93]  B. Kaminska,et al.  Minimization of the 0-1 linear programming problem under linear constraints by using neural networks: synthesis and analysis , 1996, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications.

[94]  Casimir C. Klimasauskas.  Neural networks: a new technolgy for information processing , 1989, DATB.

[95]  Mahesan Niranjan,et al.  A theoretical investigation into the performance of the Hopfield model , 1990, IEEE Trans. Neural Networks.

[96]  Petr Pavlík.  Parallel Hopfield machine , 1992, Neurocomputing.

[97]  Jun Wang,et al.  Primal and dual assignment networks , 1997, IEEE Trans. Neural Networks.

[98]  Edgar Sanchez-Sinencio,et al.  Nonlinear switched capacitor 'neural' networks for optimization problems , 1990 .