Generic Variance Bounds on Estimation and Prediction Errors in Time Series Analysis: An Entropy Perspective

In this paper, we obtain generic bounds on the variances of estimation and prediction errors in time series analysis via an information-theoretic approach. It is seen in general that the error bounds are determined by the conditional entropy of the data point to be estimated or predicted given the side information or past observations. Additionally, we discover that in order to achieve the prediction error bounds asymptotically, the necessary and sufficient condition is that the “innovation” is asymptotically white Gaussian. When restricted to Gaussian processes and 1-step prediction, our bounds are shown to reduce to the Kolmogorov-Szegö formula and Wiener-Masani formula known from linear prediction theory.

[1]  Young-Han Kim,et al.  Feedback Capacity of Stationary Gaussian Channels , 2006, 2006 IEEE International Symposium on Information Theory.

[2]  P. Graefe.  Linear stochastic systems , 1966 .

[3]  Jie Chen,et al.  An Integral Characterization of Optimal Error Covariance by Kalman Filtering , 2018, 2018 Annual American Control Conference (ACC).

[4]  Mikael Skoglund,et al.  Optimal sensor design and zero-delay source coding for continuous-time vector Gauss-Markov processes , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[5]  Shlomo Shamai,et al.  The Interplay Between Information and Estimation Measures , 2013, Found. Trends Signal Process..

[6]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[7]  Aarnout Brombacher,et al.  Probability... , 2009, Qual. Reliab. Eng. Int..

[8]  Jie Chen,et al.  Fundamental error bounds in state estimation: An information-theoretic analysis , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[9]  J. Makhoul,et al.  Linear prediction: A tutorial review , 1975, Proceedings of the IEEE.

[10]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[11]  P. P. Vaidyanathan,et al.  The Theory of Linear Prediction , 2008, Synthesis Lectures on Signal Processing.

[12]  Petros G. Voulgaris,et al.  On optimal ℓ∞ to ℓ∞ filtering , 1995, Autom..

[13]  Karl Henrik Johansson,et al.  A Frequency-Domain Characterization of Optimal Error Covariance for the Kalman-Bucy Filter , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[14]  Tryphon T. Georgiou,et al.  The Role of the Time-Arrow in Mean-Square Estimation of Stochastic Processes , 2015, IEEE Control Systems Letters.

[15]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[16]  T. Duncan.  ON THE CALCULATION OF MUTUAL INFORMATION , 1970 .

[17]  G. Picci,et al.  Linear Stochastic Systems: A Geometric Approach to Modeling, Estimation and Identification , 2016 .

[18]  Shlomo Shamai,et al.  Mutual information and minimum mean-square error in Gaussian channels , 2004, IEEE Transactions on Information Theory.

[19]  Norbert Wiener,et al.  Extrapolation, Interpolation, and Smoothing of Stationary Time Series , 1964 .

[20]  E. Parzen.  Foundations of Time Series Analysis and Prediction Theory , 2002 .

[21]  Shlomo Shamai,et al.  A View of Information-Estimation Relations in Gaussian Networks , 2017, Entropy.

[22]  Haim H. Permuter,et al.  Directed Information, Causal Estimation, and Communication in Continuous Time , 2009, IEEE Transactions on Information Theory.

[23]  Jie Chen,et al.  Towards Integrating Control and Information Theories , 2017 .

[24]  R. Gray.  Entropy and Information Theory , 1990, Springer New York.

[25]  H. W. Bode,et al.  A Simplified Derivation of Linear Least Square Smoothing and Prediction Theory , 1950, Proceedings of the IRE.

[26]  N. Wiener,et al.  The prediction theory of multivariate stochastic processes , 1957 .

[27]  Thomas Kailath,et al.  A view of three decades of linear filtering theory , 1974, IEEE Trans. Inf. Theory.

[28]  Amiel Feinstein,et al.  Information and information stability of random variables and processes , 1964 .

[29]  W. Marsden.  I and J , 2012 .

[30]  Fang Song,et al.  Generic Variance Bounds on Estimation and Prediction Errors in Time Series Analysis: An Entropy Perspective , 2019 .