A neural network theory for constrained optimization

Abstract A variety of real-world problems can be formulated into continuous optimization problems with constraint equalities. The real-world problem here can include, for example, the traveling salesman problem, the Dido’s isoperimetric problem, the Hitchcock’s transportation problem, the network flow problem and the associative memory problem. In spite of the significance, there has not yet been developed any robust solving method that works efficiently across a broad spectrum of optimization problems. The recent Hopfield’s neural network method to solve the traveling salesman problem is a potentially promising candidate because of the efficiency due to its parallel processing. His method, however, has certain drawbacks that must be removed away before it can be qualified for an efficient, robust solving method. That is: (a) locally minimum solutions instead of globally minimum; (b) possible infeasible solutions; (c) heuristic choice of network parameters and an initial state; (d) quadratic objective functions instead of arbitrary nonlinear objective functions with arbitrary nonlinear equality constraints; and (e) unorganized mathematical formulation of the network for extension. This paper develops from the Hopfield method an efficient, robust network solving method of the continuous optimization problem with constraint equalities that resolves all the drawbacks except for (a) that has already been resolved by others. The development is mathematically rigorous and thus constitutes a solid foundation of a neural network theory for constrained optimization.

[1]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[2]  Gadi Pinkas,et al.  Reasoning, Nonmonotonicity and Learning in Connectionist Networks that Capture Propositional Knowledge , 1995, Artif. Intell..

[3]  F. Fairman.  Introduction to dynamic systems: Theory, models and applications , 1979, Proceedings of the IEEE.

[4]  Gadi Pinkas,et al.  Symmetric Neural Networks and Propositional Logic Satisfiability , 1991, Neural Computation.

[5]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[6]  Andrew H. Gee,et al.  An analytical framework for optimizing neural networks , 1993, Neural Networks.

[7]  Andrew H. Gee,et al.  Polyhedral Combinatorics and Neural Networks , 1994, Neural Computation.

[8]  Kiichi Urahama,et al.  Gradient Projection Network: Analog Solver for Linearly Constrained Nonlinear Programming , 1996, Neural Computation.

[9]  E. Coddington,et al.  Theory of Ordinary Differential Equations , 1955 .

[10]  Eugene L. Lawler,et al.  Traveling Salesman Problem , 2016 .

[11]  C. Berge.  Théorie des graphes et ses applications , 1958 .

[12]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[13]  David G. Luenberger,et al.  Linear and nonlinear programming , 1984 .

[15]  Marco Budinich,et al.  A Self-Organizing Neural Network for the Traveling Salesman Problem That Is Competitive with Simulated Annealing , 1996, Neural Computation.

[16]  Behrooz Kamgar-Parsi,et al.  On problem solving with Hopfield neural networks , 1990, International 1989 Joint Conference on Neural Networks.

[17]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[19]  F. L. Hitchcock.  The Distribution of a Product from Several Sources to Numerous Localities , 1941 .

[20]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[21]  Yoshikane Takahashi.  A Unified Constructive Network Model for Problem-Solving , 1996, Theor. Comput. Sci..

[22]  Bernard Angéniol,et al.  Self-organizing feature maps and the travelling salesman problem , 1988, Neural Networks.

[23]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[24]  M. Hirsch,et al.  Differential Equations, Dynamical Systems, and Linear Algebra , 1974 .

[25]  Yoshikane Takahashi,et al.  Mathematical improvement of the Hopfield model for TSP feasible solutions by synapse dynamical systems , 1997, Neurocomputing.

[26]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[27]  Yoshikane Takahashi.  Solving Optimization Problems with Variable-Constraint by an Extended Cohen-Grossberg Model , 1996, Theor. Comput. Sci..

[28]  Gintaras V. Reklaitis,et al.  Generalized Hopfield Networks and Nonlinear Optimization , 1989, NIPS.

[29]  L. T. Grujic,et al.  Exact both construction of Lyapunov function and asymptotic stability domain determination , 1993, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[30]  Carsten Peterson,et al.  A New Method for Mapping Optimization Problems Onto Neural Networks , 1989, Int. J. Neural Syst..

[31]  Teuvo Kohonen,et al.  Associative memory. A system-theoretical approach , 1977 .

[32]  Petar D. Simic,et al.  Statistical mechanics as the underlying theory of ‘elastic’ and ‘neural’ optimisations , 1990 .

[33]  Richard Szeliski,et al.  An Analysis of the Elastic Net Approach to the Traveling Salesman Problem , 1989, Neural Computation.

[34]  Alan L. Yuille,et al.  Generalized Deformable Models, Statistical Physics, and Matching Problems , 1990, Neural Computation.

[35]  D. Koenig.  Theorie Der Endlichen Und Unendlichen Graphen , 1965 .

[36]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.