Error Stability Properties of Generalized Gradient-Type Algorithms

We present a unified framework for convergence analysis of generalized subgradient-type algorithms in the presence of perturbations. A principal novel feature of our analysis is that perturbations need not tend to zero in the limit. It is established that the iterates of the algorithms are attracted, in a certain sense, to an ɛ-stationary set of the problem, where ɛ depends on the magnitude of perturbations. Characterization of the attraction sets is given in the general (nonsmooth and nonconvex) case. The results are further strengthened for convex, weakly sharp, and strongly convex problems. Our analysis extends and unifies previously known results on convergence and stability properties of gradient and subgradient methods, including their incremental, parallel, and heavy ball modifications.

[1]  E. M. L. Beale,et al.  Nonlinear Programming: A Unified Approach. , 1970 .

[2]  N. Rouche,et al.  Stability Theory by Liapunov's Direct Method , 1977 .

[3]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[4]  F. Clarke.  Optimization And Nonsmooth Analysis , 1983 .

[5]  D. Mayne,et al.  Nondifferential optimization via adaptive smoothing , 1984 .

[6]  A. Leonov.  Approximate calculation of a pseudoinverse matrix using a generalized discrepancy principle , 1986 .

[7]  S. Zavriev.  Convergence properties of the gradient method under conditions of variable-level interference , 1990 .

[8]  Tarun Khanna,et al.  Foundations of neural networks , 1990 .

[9]  Zhi-Quan Luo,et al.  On the Convergence of the LMS Algorithm with Adaptive Learning Rate for Linear Feedforward Networks , 1991, Neural Computation.

[10]  Olvi L. Mangasarian,et al.  Backpropagation Convergence via Deterministic Nonmonotone Perturbed Minimization , 1993, NIPS.

[11]  Olvi L. Mangasarian,et al.  Mathematical Programming in Neural Networks , 1993, INFORMS J. Comput..

[12]  M. Ferris,et al.  Weak sharp minima in mathematical programming , 1993 .

[13]  Luo Zhi-quan,et al.  Analysis of an approximate gradient projection method with applications to the backpropagation algorithm , 1994 .

[14]  D. Bertsekas.  Incremental least squares methods and the extended Kalman filter , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[15]  O. Mangasarian,et al.  Serial and parallel backpropagation convergence via nonmonotone perturbed minimization , 1994 .

[16]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[17]  Petros G. Voulgaris,et al.  On optimal ℓ∞ to ℓ∞ filtering , 1995, Autom..

[18]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[19]  M. Solodov.  Convergence Analysis of Perturbed Feasible Descent Methods , 1997 .

[20]  Dimitri P. Bertsekas,et al.  A New Class of Incremental Gradient Methods for Least Squares Problems , 1997, SIAM J. Optim..

[21]  M. Solodov.  New Inexact Parallel Variable Distribution Algorithms , 1997, Comput. Optim. Appl..

[22]  Benar Fux Svaiter,et al.  Descent methods with linesearch in the presence of perturbations , 1997 .

[23]  Paul Tseng,et al.  An Incremental Gradient(-Projection) Method with Momentum Term and Adaptive Stepsize Rule , 1998, SIAM J. Optim..

[24]  Alfredo N. Iusem,et al.  On the projected subgradient method for nonsmooth convex optimization in a Hilbert space , 1998, Math. Program..

[25]  Mikhail V. Solodov,et al.  Incremental Gradient Algorithms with Stepsizes Bounded Away from Zero , 1998, Comput. Optim. Appl..