Design of Recurrent Neural Networks for Solving Constrained Least Absolute Deviation Problems

Recurrent neural networks for solving constrained least absolute deviation (LAD) problems or L1-norm optimization problems have attracted much interest in recent years. But so far most neural networks can only deal with some special linear constraints efficiently. In this paper, two neural networks are proposed for solving LAD problems with various linear constraints including equality, two-sided inequality and bound constraints. When tailored to solve some special cases of LAD problems in which not all types of constraints are present, the two networks can yield simpler architectures than most existing ones in the literature. In particular, for solving problems with both equality and one-sided inequality constraints, another network is invented. All of the networks proposed in this paper are rigorously shown to be capable of solving the corresponding problems. The different networks designed for solving the same types of problems possess the same structural complexity, which is due to the fact these architectures share the same computing blocks and only differ in connections between some blocks. By this means, some flexibility for circuits realization is provided. Numerical simulations are carried out to illustrate the theoretical results and compare the convergence rates of the networks.

[1]  Giovanni Costantini,et al.  Quasi-Lagrangian Neural Network for Convex Quadratic Optimization , 2008, IEEE Transactions on Neural Networks.

[2]  Mauro Forti,et al.  Generalized neural network for nonsmooth nonlinear programming problems , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[3]  L. Liao,et al.  A neural network for a class of convex quadratic minimax problems with constraints , 2004, IEEE Transactions on Neural Networks.

[4]  D. Kinderlehrer,et al.  An introduction to variational inequalities and their applications , 1980 .

[5]  Nicholas G. Maratos,et al.  A Nonfeasible Gradient Projection Recurrent Neural Network for Equality-Constrained Optimization Problems , 2008, IEEE Transactions on Neural Networks.

[6]  Zhishun Wang,et al.  Constrained Least Absolute Deviation Neural Networks , 2008, IEEE Transactions on Neural Networks.

[7]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[8]  Mark A. Lukas,et al.  An L 1 estimation algorithm with degeneracy and linear constraints , 2002 .

[9]  Kay Chen Tan,et al.  An asynchronous recurrent linear threshold network approach to solving the traveling salesman problem , 2008, Neurocomputing.

[10]  Xiaolin Hu,et al.  Another Simple Recurrent Neural Network for Quadratic and Linear Programming , 2009, ISNN.

[11]  Xiaolin Hu,et al.  An Alternative Recurrent Neural Network for Solving Variational Inequalities and Related Optimization Problems , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[12]  Mohamed S. Kamel,et al.  A Generalized Least Absolute Deviation Method for Parameter Estimation of Autoregressive Signals , 2008, IEEE Transactions on Neural Networks.

[13]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[14]  W. Steiger,et al.  Least Absolute Deviations: Theory, Applications and Algorithms , 1984 .

[15]  Zhishun Wang,et al.  Robust time delay estimation of bioelectric signals using least absolute deviation neural network , 2005, IEEE Transactions on Biomedical Engineering.

[16]  Xiaolin Hu,et al.  An Improved Dual Neural Network for Solving a Class of Quadratic Programming Problems and Its $k$-Winners-Take-All Application , 2008, IEEE Transactions on Neural Networks.

[17]  Carsten Peterson,et al.  Neural Networks for Optimization Problems with Inequality Constraints: The Knapsack Problem , 1993, Neural Computation.

[18]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[19]  Shuzhi Sam Ge,et al.  Design and analysis of a general recurrent neural network model for time-varying matrix inversion , 2005, IEEE Transactions on Neural Networks.

[20]  Long Cheng,et al.  A Neutral-Type Delayed Projection Neural Network for Solving Nonlinear Variational Inequalities , 2008, IEEE Transactions on Circuits and Systems II: Express Briefs.

[21]  E. Leifer,et al.  Adaptive Regression , 2005, Journal of biopharmaceutical statistics.

[22]  Jinde Cao,et al.  Solving Quadratic Programming Problems by Delayed Projection Neural Network , 2006, IEEE Transactions on Neural Networks.

[23]  Mohamed S. Kamel,et al.  Cooperative Recurrent Neural Networks for the Constrained $L_{1}$ Estimator , 2007, IEEE Transactions on Signal Processing.

[24]  N. G. Parke,et al.  Ordinary Differential Equations. , 1958 .

[25]  P. Hartman.  Ordinary Differential Equations , 1965 .

[26]  Li-Zhi Liao,et al.  A Novel Neural Network for a Class of Convex Quadratic Minimax Problems , 2006, Neural Computation.

[27]  Qingshan Liu,et al.  A One-Layer Recurrent Neural Network With a Discontinuous Hard-Limiting Activation Function for Quadratic Programming , 2008, IEEE Transactions on Neural Networks.

[28]  Long Cheng,et al.  A Recurrent Neural Network for Hierarchical Control of Interconnected Dynamic Systems , 2007, IEEE Transactions on Neural Networks.

[29]  Vassilis Zissimopoulos,et al.  Extended Hopfield models for combinatorial optimization , 1999, IEEE Trans. Neural Networks.

[30]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[31]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[32]  Xiaolin Hu,et al.  Solving Pseudomonotone Variational Inequalities and Pseudoconvex Optimization Problems Using the Projection Neural Network , 2006, IEEE Transactions on Neural Networks.

[33]  Xiaolin Hu,et al.  A New Recurrent Neural Network for Solving Convex Quadratic Programming Problems With an Application to the $k$-Winners-Take-All Problem , 2009, IEEE Transactions on Neural Networks.

[34]  Liqun Qi,et al.  A novel neural network for variational inequalities with linear and nonlinear constraints , 2005, IEEE Transactions on Neural Networks.

[35]  Xiaolin Hu,et al.  Solving Generally Constrained Generalized Linear Variational Inequalities Using the General Projection Neural Networks , 2007, IEEE Transactions on Neural Networks.

[36]  Elwood T. Olsen,et al.  L1 and L∞ minimization via a variant of Karmarkar's algorithm , 1989, IEEE Trans. Acoust. Speech Signal Process..

[37]  Xing-Bao Gao,et al.  A novel neural network for nonlinear convex programming , 2004, IEEE Trans. Neural Networks.

[38]  Wu Li,et al.  The Linear l1 Estimator and the Huber M-Estimator , 1998, SIAM J. Optim..

[39]  Jérôme Malick,et al.  A Dual Approach to Semidefinite Least-Squares Problems , 2004, SIAM J. Matrix Anal. Appl..

[40]  Youshen Xia,et al.  A recurrent neural network for nonlinear convex optimization subject to nonlinear inequality constraints , 2004, IEEE Trans. Circuits Syst. I Regul. Pap..