Estimation of Autoregressive Parameters from Noisy Observations Using Iterated Covariance Updates

Estimating the parameters of the autoregressive (AR) random process is a problem that has been well-studied. In many applications, only noisy measurements of AR process are available. The effect of the additive noise is that the system can be modeled as an AR model with colored noise, even when the measurement noise is white, where the correlation matrix depends on the AR parameters. Because of the correlation, it is expedient to compute using multiple stacked observations. Performing a weighted least-squares estimation of the AR parameters using an inverse covariance weighting can provide significantly better parameter estimates, with improvement increasing with the stack depth. The estimation algorithm is essentially a vector RLS adaptive filter, with time-varying covariance matrix. Different ways of estimating the unknown covariance are presented, as well as a method to estimate the variances of the AR and observation noise. The notation is extended to vector autoregressive (VAR) processes. Simulation results demonstrate performance improvements in coefficient error and in spectrum estimation.

[1]  W.-P. Zhu,et al.  Identification of autoregressive moving average systems from noise-corrupted observations , 2008, 2008 Joint 6th International IEEE Northeast Workshop on Circuits and Systems and TAISA Conference.

[2]  Wei Xing Zheng.  On Estimation of Autoregressive Signals in the Presence of Noise , 2006, IEEE Transactions on Circuits and Systems II: Express Briefs.

[3]  Wei Xing Zheng.  Adaptive parameter estimation of autoregressive signals from noisy observations , 1998, ICSP '98. 1998 Fourth International Conference on Signal Processing (Cat. No.98TH8344).

[4]  Lijuan Jia,et al.  On convergence of a BCLS algorithm for noisy autoregressive process estimation , 2002, Proceedings of the 41st IEEE Conference on Decision and Control, 2002..

[5]  Eric Grivel,et al.  Multichannel AR parameter estimation from noisy observations as an errors-in-variables issue , 2008, 2008 16th European Signal Processing Conference.

[6]  Tülay Adali,et al.  Unbiased RLS identification of errors-in-variables models in the presence of correlated noise , 2014, 2014 22nd European Signal Processing Conference (EUSIPCO).

[7]  J. Makhoul,et al.  Linear prediction: A tutorial review , 1975, Proceedings of the IEEE.

[8]  Petar M. Djuric,et al.  Sequential Estimation of Hidden ARMA Processes by Particle Filtering—Part II , 2017, IEEE Transactions on Signal Processing.

[9]  Yingting Luo,et al.  An Advanced Parameter Estimator of Multichannel Autoregressive Signals from Noisy Observations , 2009, 2009 International Conference on Information Engineering and Computer Science.

[10]  Stefan Werner,et al.  Recursive Total Least-Squares Algorithm Based on Inverse Power Method and Dichotomous Coordinate-Descent Iterations , 2014, IEEE Transactions on Signal Processing.

[11]  Wei Xing Zheng.  On unbiased parameter estimation of autoregressive signals observed in noise , 2003, Proceedings of the 2003 International Symposium on Circuits and Systems, 2003. ISCAS '03..

[12]  Carlos E. Davila,et al.  On the noise-compensated Yule-Walker equations , 2001, IEEE Trans. Signal Process..

[13]  Petros G. Voulgaris,et al.  On optimal ℓ∞ to ℓ∞ filtering , 1995, Autom..

[14]  K. Hasan,et al.  Multichannel autoregressive spectral estimation from noisy observations , 2000, 2000 TENCON Proceedings. Intelligent Systems and Technologies for the New Millennium (Cat. No.00CH37119).

[15]  Mohamed Najim,et al.  Consistent estimation of autoregressive parameters from noisy observations based on two interacting Kalman filters , 2006, Signal Process..

[16]  J. Cadzow.  Maximum Entropy Spectral Analysis , 2006 .

[17]  J. Cadzow,et al.  High performance spectral estimation--A new ARMA method , 1980 .

[18]  Petar M. Djuric,et al.  Sequential Estimation of Hidden ARMA Processes by Particle Filtering—Part I , 2017, IEEE Transactions on Signal Processing.

[19]  Ai-Guo Wu,et al.  Bias compensation-based recursive least-squares estimation with forgetting factors for output error moving average systems , 2014, IET Signal Process..

[20]  Mohamed A. Deriche.  AR parameter estimation from noisy data using the EM algorithm , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[21]  T. Moon,et al.  Mathematical Methods and Algorithms for Signal Processing , 1999 .

[22]  James Melsa,et al.  Least squares estimation of predictor coefficients from noisy observations , 1977, 1977 IEEE Conference on Decision and Control including the 16th Symposium on Adaptive Processes and A Special Symposium on Fuzzy Set Theory and Applications.

[23]  Jitendra K. Tugnait.  Recursive parameter estimation for noisy autoregressive signals , 1986, IEEE Trans. Inf. Theory.

[24]  R. Guidorzi,et al.  Identification of ARX models with noisy input and output , 2007, 2007 European Control Conference (ECC).

[25]  L. Scharf,et al.  A Prony method for noisy data: Choosing the signal components and selecting the order in exponential signal models , 1984, Proceedings of the IEEE.

[26]  T. Ulrych,et al.  Time series modeling and maximum entropy , 1976 .

[27]  S. Kay.  Fundamentals of statistical signal processing: estimation theory , 1993 .

[28]  T. Sen Lee.  Identification and spectral estimation of noisy multivariate autoregressive processes , 1981, ICASSP.

[29]  Albert H Nuttall,et al.  Spectral Analysis of a Univariate Process with Bad Data Points, via Maximum Entropy and Linear Predictive Techniques , 1976 .

[30]  Donald F. Gingras.  Estimation of the autoregressive parameters from observations of a noise corrupted autoregressive time series , 1982, ICASSP.

[31]  Hing Cheung So.  LMS algorithm for unbiased parameter estimation of noisy autoregressive signals , 2001 .

[32]  Luca Martino,et al.  Distributed Particle Metropolis-Hastings Schemes , 2018, 2018 IEEE Statistical Signal Processing Workshop (SSP).

[33]  Wei Xing Zheng.  On TLS estimation of autoregressive signals with noisy measurements , 2003, Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings..

[34]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[35]  Georgios B. Giannakis,et al.  Noisy input/output system identification using cumulants and the Steiglitz-McBride algorithm , 1996, IEEE Trans. Signal Process..

[36]  Yong Zhang,et al.  A modified bias compensation method for output error systems with colored noises , 2011, Proceedings of the 30th Chinese Control Conference.

[37]  Gerhard Doblinger,et al.  An adaptive Kalman filter for the enhancement of noisy AR signals , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[38]  Roberto Guidorzi,et al.  A recursive errors-in-variables method for tracking time varying autoregressive parameters from noisy observations , 2010, 2010 18th European Signal Processing Conference.

[39]  Inan Güler,et al.  AR spectral analysis of EEG signals by using maximum likelihood estimation , 2001, Comput. Biol. Medicine.

[40]  Wei Xing Zheng.  An Efficient Method for Estimation of Autoregressive Signals Subject to Colored Noise , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[41]  A. Wrench,et al.  A new approach to noise-robust LPC , 1987, ICASSP '87. IEEE International Conference on Acoustics, Speech, and Signal Processing.

[42]  M.K. Hasan,et al.  A joint technique for autoregressive spectral estimation from noisy observations , 2000, 2000 TENCON Proceedings. Intelligent Systems and Technologies for the New Millennium (Cat. No.00CH37119).

[43]  Wei Xing Zheng.  Adaptive linear prediction of autoregressive models in the presence of noise , 2000, WCC 2000 - ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000.

[44]  Sabine Van Huffel,et al.  An efficient and reliable algorithm for computing the singular subspace of a matrix, associated with its smallest singular values , 1987 .

[45]  W. X. Zheng.  Identification of Autoregressive Signals Observed in Noise , 1993, 1993 American Control Conference.

[46]  M. S. Ahmed.  Estimating the parameters of a noisy AR-process by using a bootstrap estimator , 1982, ICASSP.

[47]  Mohamed Najim,et al.  Estimation of autoregressive fading channels based on two cross-coupled H∞ filters , 2009, Signal Image Video Process..

[48]  Md. Kamrul Hasan,et al.  A new method for parameter estimation of autoregressive signals in colored noise , 2002, 2002 11th European Signal Processing Conference.

[49]  R. Lacoss.  DATA ADAPTIVE SPECTRAL ANALYSIS METHODS , 1971 .

[50]  Karthikeyan Natesan Ramamurthy,et al.  MATLAB Software for the Code Excited Linear Prediction Algorithm: The Federal Standard-1016 , 2010, MATLAB Software for the Code Excited Linear Prediction Algorithm.

[51]  E. Masry,et al.  Autoregressive spectral estimation in additive noise , 1988, IEEE Trans. Acoust. Speech Signal Process..

[52]  Yoram Bresler,et al.  Exact maximum likelihood parameter estimation of superimposed exponential signals in noise , 1986, IEEE Trans. Acoust. Speech Signal Process..

[53]  Ramdas Kumaresan,et al.  An algorithm for pole-zero modeling and spectral analysis , 1986, IEEE Trans. Acoust. Speech Signal Process..

[54]  L. Mcbride,et al.  A technique for the identification of linear systems , 1965 .

[55]  Wei Xing Zheng.  An efficient method for estimation of autoregressive signals in noise , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[56]  Elias Masry,et al.  Almost sure convergence analysis of autoregressive spectral estimation in additive noise , 1991, IEEE Trans. Inf. Theory.

[57]  Georgios B. Giannakis,et al.  Robust RLS in the Presence of Correlated Noise Using Outlier Sparsity , 2012, IEEE Transactions on Signal Processing.

[58]  Wei Xing Zheng.  Unbiased identification of autoregressive signals observed in colored noise , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[59]  Wei Xing Zheng.  An efficient algorithm for parameter estimation of noisy AR processes , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[60]  Hsiao-Chuan Wang,et al.  An extended Levinson-Durbin algorithm for the analysis of noisy autoregressive process , 1996, IEEE Signal Processing Letters.

[61]  Steven Kay.  Recursive maximum likelihood estimation of autoregressive processes , 1983 .

[62]  Takashi Yahagi,et al.  Accurate noise compensation technique for the identification of multichannel AR processes with noise , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[63]  Roberto Guidorzi,et al.  Recursive errors-in-variables approach for ar parameter estimation from noisy observations. Application to radar sea clutter rejection , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[64]  Steven Kay,et al.  The effects of noise on the autoregressive spectral estimator , 1979 .

[65]  Thomas L. Marzetta,et al.  An IQML type algorithm for AR parameter estimation from noisy covariance sequences , 2009, 2009 17th European Signal Processing Conference.

[66]  Wei Xing Zheng.  On implementation of a least-squares based algorithm for noisy autoregressive signals , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[67]  Tülay Adali,et al.  Unbiased Recursive Least-Squares Estimation Utilizing Dichotomous Coordinate-Descent Iterations , 2014, IEEE Transactions on Signal Processing.

[68]  A.V. Oppenheim,et al.  Enhancement and bandwidth compression of noisy speech , 1979, Proceedings of the IEEE.

[69]  L. Marple.  A new autoregressive spectrum analysis algorithm , 1980 .

[70]  W.-P. Zhu,et al.  A correlation domain algorithm for autoregressive system identification from noisy observations , 2008, 2008 51st Midwest Symposium on Circuits and Systems.

[71]  Kareem E. Baddour,et al.  Autoregressive modeling for fading channel simulation , 2005, IEEE Transactions on Wireless Communications.

[72]  Ramdas Kumaresan,et al.  Accurate parameter estimation of noisy speech-like signals , 1982, ICASSP.

[73]  Wei Xing Zheng.  Estimation of autoregressive signals from noisy measurements , 1997 .

[74]  Wei Xing Zheng.  A new estimation algorithm for AR signals measured in noise , 2002, 6th International Conference on Signal Processing, 2002..

[75]  J. Makhoul.  Spectral analysis of speech by linear prediction , 1973 .

[76]  R. Kumaresan,et al.  Improved spectral resolution III: Efficient realization , 1980, Proceedings of the IEEE.

[77]  Wei Xing Zheng.  A new look at parameter estimation of autoregressive signals from noisy observations , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[78]  Sabine Van Huffel,et al.  The partial total least squares algorithm , 1988 .

[79]  W. Zheng.  A least-squares based method for autoregressive signals in the presence of noise , 1999 .

[80]  Wei Xing Zheng.  An alternative method for noisy autoregressive signal estimation , 2002, 2002 IEEE International Symposium on Circuits and Systems. Proceedings (Cat. No.02CH37353).

[81]  Alimorad Mahmoudi,et al.  Parameter estimation of noisy autoregressive signals , 2010, 2010 18th Iranian Conference on Electrical Engineering.

[82]  Alexander Kukush,et al.  Measurement Error Models , 2011, International Encyclopedia of Statistical Science.

[83]  John H. L. Hansen,et al.  Discrete-Time Processing of Speech Signals , 1993 .

[84]  S. Fattah,et al.  Identification of noisy AR systems using damped sinusoidal model of autocorrelation function , 2003, IEEE Signal Processing Letters.

[85]  Ali Jamoos,et al.  Dual optimal filters for parameter estimation of a multivariate autoregressive process from noisy observations , 2011 .

[86]  R. Kumaresan,et al.  Estimation of frequencies of multiple sinusoids: Making linear prediction perform like maximum likelihood , 1982, Proceedings of the IEEE.

[87]  Joos Vandewalle,et al.  The Use of Total Linear Least Squares Techniques for Identification and Parameter Estimation , 1985 .

[88]  Luis Weruaga,et al.  Frequency-selective autoregressive estimation in noise , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[89]  A. Evans,et al.  Optimal least squares time-domain synthesis of recursive digital filters , 1973 .

[90]  Wei Xing Zheng.  Autoregressive parameter estimation from noisy data , 2000 .

[91]  S. Lawrence Marple,et al.  High resolution autoregressive spectrum analysis using noise power cancellation , 1978, ICASSP.

[92]  Wei Xing Zheng.  A fast convergent algorithm for identification of noisy autoregressive signals , 2000, 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).

[93]  Bruno Cernuschi-Frías,et al.  On the exact maximum likelihood estimation of Gaussian autoregressive processes , 1988, IEEE Trans. Acoust. Speech Signal Process..

[94]  Luis Weruaga.  Two methods for autoregressive estimationin noise , 2011, 2011 IEEE GCC Conference and Exhibition (GCC).

[95]  Steven Kay,et al.  Modern Spectral Estimation: Theory and Application , 1988 .

[96]  Wu Ai-guo,et al.  Bias compensation based recursive least squares identification for equation error models with colored noises , 2014, Proceedings of the 33rd Chinese Control Conference.

[97]  CoranderJukka,et al.  Adaptive importance sampling in signal processing , 2015 .

[98]  Ramdas Kumaresan,et al.  Improved spectral resolution II , 1980, ICASSP.

[99]  Alan V. Oppenheim,et al.  All-pole modeling of degraded speech , 1978 .

[100]  Eric Grivel,et al.  Errors-in-variables identification of noisy moving average models , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[101]  Wei Xing Zheng.  Fast adaptive identification of autoregressive signals subject to noise , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[102]  Paulo Sergio Ramirez,et al.  Fundamentals of Adaptive Filtering , 2002 .

[103]  R. Kumaresan,et al.  Improved spectral resolution , 1980, Proceedings of the IEEE.

[104]  Wei Xing Zheng,et al.  On unbiased identification of autoregressive signals with noisy measurements , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[105]  R. Kumaresan,et al.  Estimating the parameters of exponentially damped sinusoids and pole-zero modeling in noise , 1982 .

[106]  Md. Kamrul Hasan,et al.  Identification of autoregressive signals in colored noise using damped sinusoidal model , 2003 .

[107]  W.-X. Zheng.  Estimation of the parameters of autoregressive signals from colored noise-corrupted measurements , 2000, IEEE Signal Processing Letters.

[108]  James A. Cadzow.  Autoregressive Moving Average Spectral Estimation: A Model Equation Error Procedure , 1981, IEEE Transactions on Geoscience and Remote Sensing.

[109]  J. Capon.  High-resolution frequency-wavenumber spectrum analysis , 1969 .