Partitioned estimation algorithms, I: Nonlinear estimation

Abstract In a radically new approach to linear estimation, Lainiotis [33, 36–37, 52–53], using the “partition theorem”-an explicit Bayes theorem-obtained fundamentally new linear filtering and smoothing algorithms both for continuous as well as discrete data. The new algorithms are given in explicit, integral expressions of a “partitioned” form, and in terms of decoupled forward filters. The “partitioned” algorithms were shown to be especially advantageous from a computational as well as from an analysis standpoint. They are essentially based on the decomposition of the innovations into partial or conditional innovations and residuals. In this paper, the “partitioned” algorithms are shown to be the natural framework in which to study such important concepts as observability, controllability, unbiasedness, and the solution of Riccati equations. Specifically, in this paper, the “partitioned” algorithms are re-examined yielding further insight as well as several significant new results on: 1. (a) unbiased estimation and filter initialization procedures; 2. (b) stochastic observability and stochastic controllability; 3. (c) the interconnection between stochastic observability, Fisher information matrix, and the Cramer-Rao bound; 4. (d) estimation error-bounds; and most importantly 5. (e) computationally effective “partitioned” solutions of time-varying matrix Riccati equations. In fact, all of the above results have been obtained for general, time-varying, lumped, linear systems. In addition, it is shown that previously established smoothing algorithms, such as the Meditch differential algorithm and the Kailath-Frost total innovation algorithm, are readily obtained from the “partitioned” algorithms. The properties of the “partitioned” algorithms are obtained, thoroughly examined, and compared to those of other algorithms.

[1]  Jayant G. Deshpande,et al.  Optimal adaptive control: A non-linear separation theorem† , 1972 .

[2]  Walerian Kipiniak,et al.  Optimal Estimation, Identification, and Control , 1964 .

[3]  L. Schwartz,et al.  A valid mathematical model for approximate nonlinear minimal-variance filtering , 1968 .

[4]  James S. Meditch,et al.  Stochastic Optimal Linear Estimation and Control , 1969 .

[5]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[6]  Y. Ho.  On the stochastic approximation method and optimal filtering theory , 1963 .

[7]  E. Tse,et al.  A direct derivation of the optimal linear filter using the maximum principle , 1967, IEEE Transactions on Automatic Control.

[8]  M. Athans,et al.  The design of suboptimal linear time-varying systems , 1968 .

[9]  Jayant G. Deshpande,et al.  Identification and Control of Linear Stochastic Systems Using Spline Functions. , 1973 .

[10]  Thomas Kailath,et al.  Some new algorithms for recursive estimation in constant linear systems , 1973, IEEE Trans. Inf. Theory.

[11]  D. Lainiotis,et al.  A Unified Approach to Detection, Estimation, and System Identification. , 1972 .

[12]  Subrata Kumar Das,et al.  ANALYSIS OF TIME-VARYING NETWORKS , 1966 .

[13]  W. Wonham.  Some applications of stochastic difierential equations to optimal nonlinear ltering , 1964 .

[14]  Demetrios G. Lainiotis.  Sequential structure and parameter-adaptive pattern recognition-I: Supervised learning , 1970, IEEE Trans. Inf. Theory.

[15]  J. S. Meditch.  Formal algorithms for continuous-time non-linear filtering and smoothing† , 1970 .

[16]  M. Morf,et al.  Some new algorithms for recursive estimation in constant, linear, discrete-time systems , 1974 .

[17]  Daniel L. Alspach,et al.  Gaussian sum approximations for nonlinear filtering , 1970 .

[18]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[19]  Harold W. Sorenson,et al.  On the development of practical nonlinear filters , 1974, Inf. Sci..

[20]  H. Sorenson,et al.  Recursive bayesian estimation using gaussian sums , 1971 .

[21]  Thomas Kailath,et al.  Development of new estimation algorithms by innovations analysis and shift-invariance properties (Corresp.) , 1974, IEEE Trans. Inf. Theory.

[22]  T. Duncan.  PROBABILITY DENSITIES FOR DIFFUSION PROCESSES WITH APPLICATIONS TO NONLINEAR FILTERING THEORY AND DETECTION THEORY , 1967 .

[23]  D. Lainiotis,et al.  Simplified parameter quantization procedure for adaptive estimation , 1969 .

[24]  K.K. Biswas,et al.  An Approach to Fixed-Point Smoothing Problems , 1972, IEEE Transactions on Aerospace and Electronic Systems.

[25]  Jayant G. Deshpande,et al.  Parameter estimation using splines , 1974, Inf. Sci..

[26]  D. Lainiotis,et al.  Optimal state-vector estimation for non-Gaussian initial state-vector , 1971 .

[27]  Thomas Kailath,et al.  Fredholm resolvents, Wiener-Hopf equations, and Riccati differential equations , 1969, IEEE Trans. Inf. Theory.

[28]  C. Striebel,et al.  On the maximum likelihood estimates for linear dynamic systems , 1965 .

[29]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .

[30]  L. E. Zachrisson.  On optimal smoothing of continuous time Kalman processes , 1969, Inf. Sci..

[31]  E. Robinson,et al.  Recursive solution to the multichannel filtering problem , 1965 .

[32]  W. Willman,et al.  On the linear smoothing problem , 1969 .

[33]  D. Lainiotis,et al.  Monte Carlo study of the optimal non-linear estimator: linear systems with non-gaussian initial states † , 1972 .

[34]  Demetrios G. Lainiotis,et al.  Joint Detection, Estimation and System Identification , 1971, Inf. Control..

[35]  T. Kailath,et al.  An innovations approach to least-squares estimation--Part III: Nonlinear estimation in white Gaussian noise , 1971 .

[36]  Cornelius T. Leondes,et al.  Nonlinear Smoothing Theory , 1970, IEEE Trans. Syst. Sci. Cybern..

[37]  R. Mehra,et al.  On optimal and suboptimal linear smoothing , 1968 .

[38]  H. Sorenson,et al.  NONLINEAR FILTERING BY APPROXIMATION OF THE A POSTERIORI DENSITY , 1968 .

[39]  James Ting-Ho Lo,et al.  On optimal nonlinear estimation part I: Continuous observation , 1973, Inf. Sci..

[40]  H. Kushner.  Approximations to optimal nonlinear filters , 1967, IEEE Transactions on Automatic Control.

[41]  R. L. Stratonovich.  CONDITIONAL MARKOV PROCESSES , 1960 .

[42]  J. Meditch.  Orthogonal Projection and Discrete Optimal Linear Smoothing , 1967 .

[43]  H. Power,et al.  Dyadic modal control of multi-input time-invariant linear systems incorporating integral feedback , 1971 .

[44]  R. Bucy.  Nonlinear filtering theory , 1965 .

[45]  John E. Prussing,et al.  A simplified method for solving the matrix Riccati equation , 1972 .

[46]  H. Kushner.  On the dynamical equations of conditional probability density functions, with applications to optimal stochastic control theory , 1964 .

[47]  James S. Meditch.  On Optimal Linear Smoothing Theory , 1967, Inf. Control..

[48]  G. Kallianpur,et al.  Arbitrary system process with additive white noise observation errors , 1968 .

[49]  E. Stear,et al.  Optimal filtering for Gauss—Markov noise† , 1968 .

[50]  D. Lainiotis.  Optimal adaptive estimation: Structure and parameter adaptation-Part I: Linear models and continuous data , 1969 .

[51]  T. Nishimura.  A New Approach to Estimation of Initial Conditions and Smoothing Problems , 1969, IEEE Transactions on Aerospace and Electronic Systems.

[52]  H. Kushner.  Dynamical equations for optimal nonlinear filtering , 1967 .

[53]  D. C. Fraser,et al.  A new technique for the optimal smoothing of data , 1968 .

[54]  James Ting-Ho Lo,et al.  Finite-dimensional sensor orbits and optimal nonlinear filtering , 1972, IEEE Trans. Inf. Theory.

[55]  D. G. Lainiotis.  ADAPTIVE PATTERN RECOGNITION: A STATE-VARIABLE APPROACH , 1972 .

[56]  H. Kushner.  On the Differential Equations Satisfied by Conditional Probablitity Densities of Markov Processes, with Applications , 1964 .

[57]  D. Lainiotis.  A nonlinear adaptive estimation recursive algorithm , 1968 .

[58]  H. Rauch.  Solutions to the linear smoothing problem , 1963 .

[59]  Yoshikazu Sawaragi,et al.  State estimation for continuous-time system with interrupted observation , 1973, CDC 1973.

[60]  Rangasami Sridhar,et al.  Sequential estimation of states and parameters in noisy non-linear dynamical systems , 1966 .

[61]  D. Lainiotis.  Optimal linear smoothing : Continuous data case † , 1973 .

[62]  H. Kwakernaak,et al.  Optimal filtering in linear systems with time delays , 1967, IEEE Transactions on Automatic Control.

[63]  Demetrios G. Lainiotis,et al.  Discrete Riccati equation solutions: Partitioned algorithms , 1975 .

[64]  H. Sorenson,et al.  Nonlinear Bayesian estimation using Gaussian sum approximations , 1972 .

[65]  D. Magill.  Optimal adaptive estimation of sampled stochastic processes , 1965 .

[66]  D. Lainiotis.  Optimal adaptive estimation: Structure and parameter adaption , 1971 .

[67]  N. Levinson.  The Wiener (Root Mean Square) Error Criterion in Filter Design and Prediction , 1946 .

[68]  R. Bucy,et al.  Digital synthesis of non-linear filters , 1971 .

[69]  T. Kailath,et al.  An innovations approach to least-squares estimation--Part II: Linear smoothing in additive white noise , 1968 .

[70]  D. Lainiotis,et al.  On joint detection, estimation and system identification: discrete data case† , 1973 .

[71]  D. G. Lainiotis,et al.  Optimal adaptive filter realizations for sample stochastic processes with an unknown parameter , 1969 .

[72]  Thomas Kailath,et al.  A general likelihood-ratio formula for random signals in Gaussian noise , 1969, IEEE Trans. Inf. Theory.

[73]  E. Stear,et al.  Near-optimal non-linear filtering using quasi-moment functions† , 1970 .

[74]  Brian D. O. Anderson,et al.  Smoothing as an improvement on filtering: a universal bound , 1971 .

[75]  D. G. Lainiotis,et al.  LEARNING SYSTEMS FOR MINIMUM RISK ADAPTIVE PATTERN CLASSIFICATION AND OPTIMAL ADAPTIVE ESTIMATION. , 1967 .

[76]  D. Lainiotis.  Optimal non-linear estimation† , 1971 .

[77]  Brian D. O. Anderson,et al.  New linear smoothing formulas , 1972 .

[78]  James S. Meditch.  Optimal fixed-point continuous linear smoothing , 1966 .

[79]  John M. Richardson.  The implicit conditioning method in statistical mechanics , 1974, Inf. Sci..

[80]  Henry Cox,et al.  On the estimation of state variables and parameters for noisy dynamic systems , 1964 .

[81]  David L Kleinman,et al.  Suboptimal design of linear regulator systems subject to computer storage limitations , 1967 .

[82]  D. Lainiotis.  Optimal adaptive estimation: Structure and parameter adaptation , 1970 .

[83]  D. Lainiotis.  Optimal nonlinear estimation , 1971, CDC 1971.

[84]  A. Lindquist.  A New Algorithm for Optimal Filtering of Discrete-Time Stationary Processes , 1974 .

[85]  Y. C. Ho,et al.  The Method of Least Squares and Optimal Filtering Theory , 1962 .

[86]  Demetrios G. Lainiotis,et al.  Optimal Estimation in the Presence of Unknown Parameters , 1969, IEEE Trans. Syst. Sci. Cybern..

[87]  Demetrios G. Lainiotis,et al.  Partitioned linear estimation algorithms: Discrete case , 1975 .

[88]  R. Mortensen.  Maximum-likelihood recursive nonlinear filtering , 1968 .

[89]  David Q. Mayne,et al.  A solution of the smoothing problem for linear dynamic systems , 1966, Autom..

[90]  Y. Ho,et al.  A Bayesian approach to problems in stochastic estimation and control , 1964 .

[91]  R. E. Kalman,et al.  FUNDAMENTAL STUDY OF ADAPTIVE CONTROL SYSTEMS , 1962 .