论文信息 - September 1994 LIDS-P-2267 A HYBRID INCREMENTAL GRADIENT METHOD 1 FOR LEAST SQUARES PROBLEMS

September 1994 LIDS-P-2267 A HYBRID INCREMENTAL GRADIENT METHOD 1 FOR LEAST SQUARES PROBLEMS

The LMS method for linear least squares problems differs from the steepest descent method in that it processes data blocks one-by-one, with intermediate adjustment of the parameter vector under optimization. This mode of operation often leads to faster convergence when far from the eventual limit, and to slower (sublinear) convergence when close to the optimal solution. We embed both LMS and steepest descent, as well as other intermediate methods, within a oneparameter class of algorithms, and we propose a hybrid method that combines the faster early convergence rate of LMS with the faster ultimate linear convergence rate of steepest descent. 1 Research supported by NSF under Grant 9300494-DMI. 2 Department of Electrical Engineering and Computer Science, M.I.T., Cambridge, Mass., 02139.1

D. Bertsekas

[1] Xingshi He,et al. Introduction to Optimization , 2019, Optimization for Chemical and Biochemical Engineering.

[2] Olvi L. Mangasarian,et al. Mathematical Programming in Neural Networks , 1993, INFORMS J. Comput..

[3] Zhi-Quan Luo,et al. On the Convergence of the LMS Algorithm with Adaptive Learning Rate for Linear Feedforward Networks , 1991, Neural Computation.

[4] H. White. Some Asymptotic Results for Learning in Single Hidden-Layer Feedforward Network Models , 1989 .

[5] John N. Tsitsiklis,et al. Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[6] Luigi Grippo,et al. A class of unconstrained minimization methods for neural network training , 1994 .

[7] O. Mangasarian,et al. Serial and parallel backpropagation convergence via nonmonotone perturbed minimization , 1994 .