Neurodynamical Optimization

Dynamical (or ode) system and neural network approaches for optimization have been co-existed for two decades. The main feature of the two approaches is that a continuous path starting from the initial point can be generated and eventually the path will converge to the solution. This feature is quite different from conventional optimization methods where a sequence of points, or a discrete path, is generated. Even dynamical system and neural network approaches share many common features and structures, yet a complete comparison for the two approaches has not been available. In this paper, based on a detailed study on the two approaches, a new approach, termed neurodynamical approach, is introduced. The new neurodynamical approach combines the attractive features in both dynamical (or ode) system and neural network approaches. In addition, the new approach suggests a systematic procedure and framework on how to construct a neurodynamical system for both unconstrained and constrained problems. In analyzing the stability issues of the underlying dynamical (or ode) system, the neurodynamical approach adopts a new strategy, which avoids the Lyapunov function. Under the framework of this neurodynamical approach, strong theoretical results as well as promising numerical results are obtained.

[1]  Bingsheng He,et al.  Inexact implicit methods for monotone general variational inequalities , 1999, Math. Program..

[2]  P. Tseng,et al.  Modified Projection-Type Methods for Monotone Variational Inequalities , 1996 .

[3]  Andrzej Cichocki,et al.  A new neural network for solving linear programming problems , 1996 .

[4]  Robert Schaback,et al.  An extended continuous Newton method , 1990 .

[5]  Jun Wang,et al.  A general methodology for designing globally convergent optimization neural networks , 1998, IEEE Trans. Neural Networks.

[6]  Z. R. Novaković.  Solving systems of non-linear equations using the Lyapunov direct method , 1990 .

[7]  Vitali G. Zhadan,et al.  A relaxation method for solving problems of non-linear programming , 1977 .

[8]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[9]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[10]  C. Botsaris,et al.  A curvilinear optimization method based upon iterative estimation of the eigensystem of the Hessian matrix , 1978 .

[11]  Leon O. Chua,et al.  Neural networks for nonlinear programming , 1988 .

[12]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[13]  Kok Lay Teo,et al.  Gradient-flow approach for computing a nonlinear-quadratic optimal-output feedback gain matrix , 1995 .

[14]  C. Botsaris.  Differential gradient methods , 1978 .

[15]  J. Willems,et al.  Stability theory of dynamical systems , 1970 .

[16]  P. Boggs.  The Solution of Nonlinear Operator Equations by A-stable Integration Techniques , 1970 .

[17]  Li-Zhi Liao,et al.  Solving nonlinear complementarity problems with neural networks: a reformulation method approach , 2001 .

[18]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[19]  M. C. Bartholomew-Biggs,et al.  ODE versus SQP methods for constrained optimization , 1989 .

[20]  Liqun Qi,et al.  Stability Analysis of Gradient-Based Neural Networks for Optimization Problems , 2001, J. Glob. Optim..

[21]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[22]  Zeng-Guang Hou,et al.  A neural network for hierarchical optimization of nonlinear large-scale systems , 1998, Int. J. Syst. Sci..

[23]  Bin-Xin He.  Solving a class of linear projection equations , 1994 .

[24]  Edgar Sanchez-Sinencio,et al.  Nonlinear switched capacitor 'neural' networks for optimization problems , 1990 .

[25]  W K Chen,et al.  A high-performance neural network for solving linear and quadratic programming problems , 1996, IEEE Trans. Neural Networks.

[26]  S. Fang,et al.  Solving convex programming problems with equality constraints by neural networks , 1998 .

[27]  B. Goh,et al.  Trajectory-following algorithms for min-max optimization problems , 1992 .

[28]  K. Tanabe.  A geometric method in nonlinear programming , 1980 .

[29]  Kurt M. Anstreicher,et al.  Linear programming and the newton barrier flow , 1988, Math. Program..

[30]  S. Żak,et al.  Sliding Modes in Solving Convex Programming Problems , 1998 .

[31]  Liao Li-Zhi,et al.  A neural network for the linear complementarity problem , 1999 .

[32]  M. Bartholomew-Biggs,et al.  Some effective methods for unconstrained optimization based on the solution of systems of ordinary differential equations , 1989 .

[33]  C. Botsaris,et al.  A class of methods for unconstrained minimization based on stable numerical integration techniques , 1978 .

[34]  Francesco Zirilli,et al.  Algorithm 617: DAFNE: a differential-equations algorithm for nonlinear equations , 1984, ACM Trans. Math. Softw..

[35]  Yong Shi,et al.  An ODE method of solving nonlinear programming , 1997 .

[36]  M. Chu.  On the Continuous Realization of Iterative Processes , 1988 .

[37]  Francesco Zirilli,et al.  A differential-equations algorithm for nonlinear equations , 1984, ACM Trans. Math. Softw..

[38]  B. He.  A class of projection and contraction methods for monotone variational inequalities , 1997 .

[39]  A. Goldstein.  Convex programming in Hilbert space , 1964 .

[40]  Malur K. Sundareshan,et al.  Exponential stability and a systematic synthesis of a neural network for quadratic minimization , 1991, Neural Networks.

[41]  Olvi L. Mangasarian,et al.  Mathematical Programming in Neural Networks , 1993, INFORMS J. Comput..

[42]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[43]  Shengwei Zhang,et al.  Lagrange programming neural networks , 1992 .

[44]  Leon O. Chua,et al.  Nonlinear programming without computation , 1984 .

[45]  Abdesselam Bouzerdoum,et al.  Neural network for quadratic optimization with bound constraints , 1993, IEEE Trans. Neural Networks.

[46]  D. Jacobson,et al.  A Newton-type curvilinear search method for optimization , 1976 .

[47]  Bingsheng He,et al.  A neural network model for monotone linear asymmetric variational inequalities , 2000, IEEE Trans. Neural Networks Learn. Syst..

[48]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[49]  Stefen Hui,et al.  On solving constrained optimization problems with neural networks: a penalty method approach , 1993, IEEE Trans. Neural Networks.

[50]  Youshen Xia.  A new neural network for solving linear programming problems and its application , 1996, IEEE Trans. Neural Networks.

[51]  Hiroshi Yamashita,et al.  A differential equation approach to nonlinear programming , 1980, Math. Program..

[52]  F. H. Branin.  Widely convergent method for finding multiple solutions of simultaneous nonlinear equations , 1972 .

[53]  Youshen Xia,et al.  A new neural network for solving linear and quadratic programming problems , 1996, IEEE Trans. Neural Networks.

[54]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[55]  V. Parisi,et al.  A New Method for Solving Nonlinear Simultaneous Equations , 1979 .

[56]  Michael A. Shanblatt,et al.  A two-phase optimization neural network , 1992, IEEE Trans. Neural Networks.

[57]  Michael A. Shanblatt,et al.  Linear and quadratic programming neural network analysis , 1992, IEEE Trans. Neural Networks.

[58]  Elizabeth Eskow,et al.  A New Modified Cholesky Factorization , 1990, SIAM J. Sci. Comput..

[59]  S. Schäffler,et al.  A trajectory-following method for unconstrained optimization , 1990 .

[60]  J. Snyman.  A new and dynamic method for unconstrained minimization , 1982 .

[61]  Boris Polyak,et al.  Constrained minimization methods , 1966 .

[62]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[63]  N. Hassan,et al.  An ordinary differential equation in nonlinear programming , 1990 .

[64]  Yong Shi,et al.  A Convergence of ODE Method in Constrained Optimization , 1998 .

[65]  Stefen Hui,et al.  Solving linear programming problems with neural networks: a comparative study , 1995, IEEE Trans. Neural Networks.