Twenty-one ML estimators for model selection

Classical approaches to determine a suitable model structure from observed input-output data are based on hypothesis tests and information-based criteria. Recently, the model structure has been considered as a stochastic variable, and standard estimation techniques have been proposed. The resulting estimators are closely related to the aforementioned methods. However, it turns out that there are a number of prior choices in the problem formulation, which are crucial for the estimators' behavior. The contribution of this paper is to clarify the role of the prior choices, to examine a number of possibilities and to show which estimators are consistent. This is done in a linear regression framework. For autoregressive models, we also investigate a novel prior assumption on stability, and give the estimator for the model order and the parameters themselves.

[1]  D. Lainiotis,et al.  System identification : advances and case studies , 1976 .

[2]  H. Akaike.  Fitting autoregressive models for prediction , 1969 .

[3]  Lee D. Davisson,et al.  The prediction error of stationary Gaussian time series of unknown covariance , 1965, IEEE Trans. Inf. Theory.

[4]  B. G. Quinn,et al.  The determination of the order of an autoregression , 1979 .

[5]  B. Friedlander,et al.  Lattice filters for adaptive processing , 1982, Proceedings of the IEEE.

[6]  L. Ahlfors.  Complex Analysis , 1979 .

[7]  E. Hannan,et al.  Recursive estimation of mixed autoregressive-moving average order , 1982 .

[8]  J. Rissanen.  Estimation of structure by minimum description length , 1982 .

[9]  C. Z. Wei.  On Predictive Least Squares Principles , 1992 .

[10]  Elkan F. Halpern.  Polynomial Regression from a Bayesian Approach , 1973 .

[11]  Hirotugu Akaike,et al.  On entropy maximization principle , 1977 .

[12]  Pieter Eykhoff,et al.  Trends and progress in system identification , 1981 .

[13]  László Gerencsér,et al.  AR(infty) estimation and nonparametric stochastic complexity , 1992, IEEE Trans. Inf. Theory.

[14]  H. Akaike.  Canonical Correlation Analysis of Time Series and the Use of an Information Criterion , 1976 .

[15]  Sándor M. Veres,et al.  Structure selection of stochastic dynamic systems : the information criterion approach , 1991 .

[16]  E. Hannan.  The Estimation of the Order of an ARMA Process , 1980 .

[17]  H. Jeffreys,et al.  Theory of probability , 1896 .

[18]  Hirotugu Akaike,et al.  MODERN DEVELOPMENT OF STATISTICAL METHODS , 1981 .

[19]  A. Atkinson.  Posterior probabilities for choosing a regression model , 1978 .

[20]  Lennart Ljung,et al.  Theory and Practice of Recursive Identification , 1983 .

[21]  Petre Stoica,et al.  Decentralized Control , 2018, The Control Systems Handbook.

[22]  Robert H. Shumway,et al.  Improved estimators of Kullback-Leibler information for autoregressive model selection in small samples , 1990 .

[23]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[24]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[25]  S. Veres.  Relations between information criteria for model-structure selection Part 1. The role of bayesian model order estimation , 1990 .

[26]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[27]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[28]  E. J. Hannan,et al.  Estimating the dimension of a linear system , 1981 .

[29]  László Gerencsér,et al.  A Computable Criterion for Model Selection for Linear Stochastic Systems , 1992 .

[30]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[31]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[32]  H. Akaike.  The Interpretation of Improper Prior Distributions as Limits of Data Dependent Proper Prior Distributions , 1980 .

[33]  R. Shibata.  Asymptotically Efficient Selection of the Order of the Model for Estimating Parameters of a Linear Process , 1980 .

[34]  J. Rissanen.  Shortest Data Description and Consistency of Order Estimates in Arma-Processes , 1979 .

[35]  C. L. Mallows.  Some comments on C_p , 1973 .

[36]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[37]  S. D. Hill,et al.  Least-informative Bayesian prior distributions for finite samples based on information theory , 1990 .

[38]  R. Shibata.  Selection of the order of an autoregressive model by Akaike's information criterion , 1976 .

[39]  Donald Poskitt,et al.  On the posterior odds of time series models , 1983 .