Incremental Least Squares Methods and the Extended Kalman Filter

In this paper we propose and analyze nonlinear least squares methods which process the data incrementally, one data block at a time. Such methods are well suited for large data sets and real time operation and have received much attention in the context of neural network training problems. We focus on the extended Kalman filter, which may be viewed as an incremental version of the Gauss--Newton method. We provide a nonstochastic analysis of its convergence properties, and we discuss variants aimed at accelerating its convergence.

[1]  R. Mehra.  A comparison of several nonlinear filters for reentry vehicle tracking , 1971 .

[2]  Robert R. Tenney,et al.  A tracking filter for maneuvering sources , 1977 .

[3]  Harold J. Kushner,et al.  wchastic. approximation methods for constrained and unconstrained systems , 1978 .

[4]  John B. Moore,et al.  Improved Extended Kalman Filter Design for Passive Tracking , 1979 .

[5]  B. Ursin.  Asymptotic convergence properties of the extended Kalman filter using filtered state estimates , 1980 .

[6]  T. Westerlund,et al.  Remarks on "Asymptotic behavior of the extended Kalman filter as a parameter estimator for linear systems" , 1980 .

[7]  D. Bertsekas.  Projected Newton methods for optimization problems with simple constraints , 1981, 1981 20th IEEE Conference on Decision and Control including the Symposium on Adaptive Processes.

[8]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[9]  John N. Tsitsiklis,et al.  Problems in decentralized decision making and computation , 1984 .

[10]  John N. Tsitsiklis,et al.  Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms , 1984, 1984 American Control Conference.

[11]  S. Thomas Alexander,et al.  Adaptive Signal Processing , 1986, Texts and Monographs in Computer Science.

[12]  H. White.  Some Asymptotic Results for Learning in Single Hidden-Layer Feedforward Network Models , 1989 .

[13]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[14]  Halbert White,et al.  Learning in Artificial Neural Networks: A Statistical Perspective , 1989, Neural Computation.

[15]  Spyros G. Tzafestas,et al.  Learning algorithms for neural networks with the Kalman filters , 1990, J. Intell. Robotic Syst..

[16]  Zhi-Quan Luo,et al.  On the Convergence of the LMS Algorithm with Adaptive Learning Rate for Linear Feedforward Networks , 1991, Neural Computation.

[17]  Mark E. Oxley,et al.  Comparative Analysis of Backpropagation and the Extended Kalman Filter for Training Multilayer Perceptrons , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Olvi L. Mangasarian,et al.  Mathematical Programming in Neural Networks , 1993, INFORMS J. Comput..

[19]  Bradley M. Bell,et al.  The Iterated Kalman Smoother as a Gauss-Newton Method , 1994, SIAM J. Optim..

[20]  Luo Zhi-quan,et al.  Analysis of an approximate gradient projection method with applications to the backpropagation algorithm , 1994 .

[21]  Luigi Grippo,et al.  A class of unconstrained minimization methods for neural network training , 1994 .

[22]  O. Mangasarian,et al.  Serial and parallel backpropagation convergence via nonmonotone perturbed minimization , 1994 .

[23]  Petros G. Voulgaris,et al.  On optimal ℓ∞ to ℓ∞ filtering , 1995, Autom..

[24]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[25]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[26]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .