A review of the Expectation Maximization algorithm in data-driven process identification

Abstract The Expectation Maximization (EM) algorithm has been widely used for parameter estimation in data-driven process identification. EM is an algorithm for maximum likelihood estimation of parameters and ensures convergence of the likelihood function. In presence of missing variables and in ill conditioned problems, EM algorithm greatly assists the design of more robust identification algorithms. Such situations frequently occur in industrial environments. Missing observations due to sensor malfunctions, multiple process operating conditions and unknown time delay information are some of the examples that can resort to the EM algorithm. In this article, a review on applications of the EM algorithm to address such issues is provided. Future applications of EM algorithm as well as some open problems are also provided.

[1]  Bhavik R. Bakshi,et al.  Representation of process trends—III. Multiscale extraction of trends from process data , 1994 .

[2]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[3]  F. Ding,et al.  Maximum likelihood Newton recursive and the Newton iterative estimation algorithms for Hammerstein CARAR systems , 2014 .

[4]  F. Ding,et al.  The filtering based maximum likelihood recursive least squares estimation for multiple-input single-output systems ☆ , 2016 .

[5]  Hui Zhang,et al.  Modified student's t-hidden Markov model for pattern recognition and classification , 2013, IET Signal Process..

[6]  Alireza Fatehi,et al.  A Data-Driven Hybrid ARX and Markov Chain Modeling Approach to Process Identification With Time-Varying Time Delays , 2017, IEEE Transactions on Industrial Electronics.

[7]  Thomas B. Schön,et al.  System identification of nonlinear state-space models , 2011, Autom..

[8]  Steven X. Ding,et al.  A Review on Basic Data-Driven Approaches for Industrial Process Monitoring , 2014, IEEE Transactions on Industrial Electronics.

[9]  Christopher M. Bishop,et al.  Robust Bayesian Mixture Modelling , 2005, ESANN.

[10]  N M Laird,et al.  Maximum Likelihood Analysis of Logistic Regression Models with Incomplete Covariate Data and Auxiliary Information , 2001, Biometrics.

[11]  Sirish L. Shah,et al.  Treatment of missing values in process data analysis , 2008 .

[12]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[13]  Zhen Zhang,et al.  Maximum likelihood estimation method for dual-rate Hammerstein systems , 2017 .

[14]  R. Shumway,et al.  AN APPROACH TO TIME SERIES SMOOTHING AND FORECASTING USING THE EM ALGORITHM , 1982 .

[15]  Jean-Pierre Richard,et al.  Time-delay systems: an overview of some recent advances and open problems , 2003, Autom..

[16]  R. Pearson,et al.  Gray-box identification of block-oriented nonlinear models , 2000 .

[17]  Zhiqiang Ge,et al.  Dynamic mixture probabilistic PCA classifier modeling and application for fault classification , 2015 .

[18]  Theodora Kourti,et al.  Troubleshooting of an Industrial Batch Process Using Multivariate Methods , 2003 .

[19]  Hamid Reza Karimi,et al.  Identification of LTI Time-Delay Systems with Missing Output Data Using GEM Algorithm , 2014 .

[20]  Raghunathan Rengaswamy,et al.  A review of process fault detection and diagnosis: Part III: Process history based methods , 2003, Comput. Chem. Eng..

[21]  Myong Kee Jeong,et al.  Robust Probabilistic Multivariate Calibration Model , 2008, Technometrics.

[22]  Ahmet Palazoglu,et al.  Classification of process trends based on fuzzified symbolic representation and hidden Markov models , 1998 .

[23]  Si-Zhao Joe Qin,et al.  Survey on data-driven industrial process monitoring and diagnosis , 2012, Annu. Rev. Control..

[24]  In-Beum Lee,et al.  Fault Detection Based on a Maximum-Likelihood Principal Component Analysis (PCA) Mixture , 2005 .

[25]  Niclas Bergman,et al.  Recursive Bayesian Estimation : Navigation and Tracking Applications , 1999 .

[26]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[27]  Nima Sammaknejad,et al.  Approaches to robust process identification: A review and tutorial of probabilistic methods , 2018, Journal of Process Control.

[28]  Michael Athans,et al.  Guaranteed properties of gain scheduled control for linear parameter-varying plants , 1991, Autom..

[29]  Zhi-huan Song,et al.  Mixture Bayesian regularization method of PPCA for multimode process monitoring , 2010 .

[30]  Biao Huang,et al.  Identification of switched Markov autoregressive eXogenous systems with hidden switching state , 2012, Autom..

[31]  René Vidal,et al.  Recursive identification of switched ARX systems , 2008, Autom..

[32]  B. Leroux.  Consistent estimation of a mixing distribution , 1992 .

[33]  David A. Landgrebe,et al.  Robust parameter estimation for mixture model , 2000, IEEE Trans. Geosci. Remote. Sens..

[34]  Wei Wang,et al.  Stability Analysis for Linear Switched Systems With Time-Varying Delay , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[35]  Shengyuan Xu,et al.  Quadratic stability and stabilization of uncertain linear discrete-time systems with state delay , 2001, Syst. Control. Lett..

[36]  Te-Won Lee,et al.  On the multivariate Laplace distribution , 2006, IEEE Signal Processing Letters.

[37]  F. Ding,et al.  Maximum likelihood gradient-based iterative estimation algorithm for a class of input nonlinear controlled autoregressive ARMA systems , 2015 .

[38]  Zhiqiang Ge,et al.  HMM-Driven Robust Probabilistic Principal Component Analyzer for Dynamic Process Fault Classification , 2015, IEEE Transactions on Industrial Electronics.

[39]  J. Tukey.  The Future of Data Analysis , 1962 .

[40]  F. Diebold,et al.  Regime Switching with Time-Varying Transition Probabilities , 2020, Business Cycles.

[41]  Biao Huang,et al.  Multiple-Model Based Linear Parameter Varying Time-Delay System Identification with Missing Output Data Using an Expectation-Maximization Algorithm , 2014 .

[42]  R. K. pearson,et al.  Control Systems, Identification , 2003 .

[43]  Jean X. Gao,et al.  Probabilistic Partial Least Square Regression: A Robust Model for Quantitative Analysis of Raman Spectroscopy Data , 2011, 2011 IEEE International Conference on Bioinformatics and Biomedicine.

[44]  Biao Huang,et al.  A novel approach to process operating mode diagnosis using conditional random fields in the presence of missing data , 2018, Comput. Chem. Eng..

[45]  Karolos M. Grigoriadis,et al.  LPV Systems with parameter-varying time delays: analysis and control , 2001, Autom..

[46]  G. Stephanopoulos,et al.  Representation of process trends—Part II. The problem of scale and qualitative scaling , 1990 .

[47]  Furong Gao,et al.  Review of Recent Research on Data-Based Process Monitoring , 2013 .

[48]  Alireza Fatehi,et al.  Operating condition diagnosis based on HMM with adaptive transition probabilities in presence of missing observations , 2015 .

[49]  Theodora Kourti,et al.  Statistical Process Control of Multivariate Processes , 1994 .

[50]  Alireza Fatehi,et al.  Adaptive monitoring of the process operation based on symbolic episode representation and hidden Markov models with application toward an oil sand primary separation , 2014, Comput. Chem. Eng..

[51]  Zhiqiang Ge,et al.  Semi-supervised PLVR models for process monitoring with unequal sample sizes of process variables and quality variables , 2015 .

[52]  Biao Huang,et al.  Multiple model approach to nonlinear system identification with an uncertain scheduling variable using EM algorithm , 2013 .

[53]  Biao Huang,et al.  Identification of nonlinear parameter varying systems with missing output data , 2012 .

[54]  Jay H. Lee,et al.  Realistic disturbance modeling using Hidden Markov Models: Applications in model-based process control , 2009 .

[55]  Ahmet Palazoglu,et al.  Classification of abnormal plant operation using multiple process variable trends , 2001 .

[56]  Bassam Bamieh,et al.  Identification of linear parameter varying models , 2002 .

[57]  Stéphane Lecoeuche,et al.  A recursive identification algorithm for switched linear/affine models , 2011 .

[58]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[59]  Dongqing Wang,et al.  Recursive maximum likelihood identification method for a multivariable controlled autoregressive moving average system , 2016, IMA J. Math. Control. Inf..

[60]  Kazushi Nakano,et al.  RBF-ARX model based nonlinear system modeling and predictive control with application to a NOx decomposition process , 2004 .

[61]  Sean Borman,et al.  The Expectation Maximization Algorithm A short tutorial , 2006 .

[62]  Johannes Blömer,et al.  Adaptive Seeding for Gaussian Mixture Models , 2013, PAKDD.

[63]  Dimitris Karlis,et al.  Choosing Initial Values for the EM Algorithm for Finite Mixtures , 2003, Comput. Stat. Data Anal..

[64]  Arnaud Doucet,et al.  Particle filters for state estimation of jump Markov linear systems , 2001, IEEE Trans. Signal Process..

[65]  Michael I. Jordan,et al.  On Convergence Properties of the EM Algorithm for Gaussian Mixtures , 1996, Neural Computation.

[66]  Yucai Zhu,et al.  Nonlinear MPC using an identified LPV model , 2009 .

[67]  Mohamed Darouach,et al.  Linear functional observers for systems with delays in state variables , 2001, IEEE Trans. Autom. Control..

[68]  P.L.D. Peres,et al.  Robust H/sub /spl infin// filtering for uncertain linear systems with multiple time-varying state delays: an LMI approach , 1999, Proceedings of the 38th IEEE Conference on Decision and Control (Cat. No.99CH36304).

[69]  Zhiqiang Ge,et al.  Probabilistic latent variable regression model for process-quality monitoring , 2014 .

[70]  R. P. Marques,et al.  Discrete-Time Markov Jump Linear Systems , 2004, IEEE Transactions on Automatic Control.

[71]  A. Doucet.  On sequential Monte Carlo methods for Bayesian filtering , 1998 .

[72]  Feng Ding,et al.  Maximum Likelihood Recursive Least Squares Estimation for Multivariable Systems , 2014, Circuits Syst. Signal Process..

[73]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[74]  Biao Huang,et al.  Robust identification of piecewise/switching autoregressive exogenous process , 2009 .

[75]  W. R. Schucany,et al.  A Comparison of Minimum Distance and Maximum Likelihood Estimation of a Mixture Proportion , 1984 .

[76]  Biao Huang,et al.  Multiple model LPV approach to nonlinear process identification with EM algorithm , 2011 .

[77]  James D. Hamilton.  Analysis of time series subject to changes in regime , 1990 .

[78]  J. Schafer,et al.  Missing data: our view of the state of the art. , 2002, Psychological methods.

[79]  Y. Arkun,et al.  Estimation of nonlinear systems using linear multiple models , 1997 .

[80]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[81]  V. Lunel,et al.  Identification problems in functional differential equations , 1997, Proceedings of the 36th IEEE Conference on Decision and Control.

[82]  Jeff A. Bilmes,et al.  A gentle tutorial of the em algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models , 1998 .

[83]  Yaojie Lu,et al.  Robust multiple-model LPV approach to nonlinear process identification using mixture t distributions , 2014 .

[84]  Pin-Lin Liu,et al.  Robust exponential stability for uncertain time-varying delay systems with delay dependence , 2009, J. Frankl. Inst..

[85]  Peng Shi,et al.  Robust Fault Detection for Switched Linear Systems With State Delays , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[86]  Geoffrey J. McLachlan.  On the choice of starting values for the EM algorithm in fitting mixture models , 1988 .

[87]  R. Casey,et al.  Advances in Pattern Recognition , 1971 .

[88]  R. Morales-Menendez,et al.  Estimation and control of industrial processes with particle filters , 2003, Proceedings of the 2003 American Control Conference, 2003..

[89]  Biao Huang,et al.  FIR model identification of multirate processes with random delays using EM algorithm , 2013 .

[90]  B. Lindsay,et al.  The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family , 1994 .

[91]  David S. Stoffer,et al.  Time series analysis and its applications , 2000 .

[92]  M. Nazmul Karim,et al.  Model-predictive pH control using real-time NARX approach , 1994 .

[93]  Zoubin Ghahramani,et al.  Optimization with EM and Expectation-Conjugate-Gradient , 2003, ICML.

[94]  A. A. Bachnas,et al.  A review on data-driven linear parameter-varying modeling approaches: A high-purity distillation column case study , 2014 .

[95]  Yaojie Lu,et al.  Robust Diagnosis of Operating Mode Based on Time-Varying Hidden Markov Models , 2016, IEEE Transactions on Industrial Electronics.

[96]  Shesh N. Rai,et al.  Improving the EM Algorithm , 1993 .

[97]  Sotirios Chatzis,et al.  Robust Sequential Data Modeling Using an Outlier Tolerant Hidden Markov Model , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[98]  Brian F. Feeny,et al.  NONLINEAR SYSTEM IDENTIFICATION , 2001 .

[99]  Alfredo Germani,et al.  A state observer for nonlinear delay systems , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[100]  R. B. Gopaluni.  A particle filter approach to identification of nonlinear processes under missing observations , 2008 .

[101]  S. Qin,et al.  Multimode process monitoring with Bayesian inference‐based finite Gaussian mixture models , 2008 .

[102]  G. Stephanopoulos,et al.  Representation of process trends—Part I. A formal representation framework , 1990 .

[103]  Biao Huang,et al.  Development and industrial application of soft sensors with on-line Bayesian model updating strategy , 2013 .

[104]  Feng Ding,et al.  Data filtering based multi-innovation extended gradient method for controlled autoregressive autoregressive moving average systems using the maximum likelihood principle , 2017, Math. Comput. Simul..

[105]  Sheetal Kalyani,et al.  Robust Statistics Based Expectation-Maximization Algorithm for Channel Tracking in OFDM Systems , 2007, 2007 IEEE International Conference on Communications.

[106]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[107]  Sotirios Chatzis,et al.  A variational Bayesian methodology for hidden Markov models utilizing Student's-t mixtures , 2011, Pattern Recognit..