A Comparative Study of Information Criteria for Model Selection

To build good models, we need to know the appropriate model size. To handle this problem, a variety of information criteria have already been proposed, each with a different background. In this paper, we consider the problem of model selection and investigate the performance of a number of proposed information criteria and whether the assumption to obtain the formulae that fitting errors are normally distributed hold or not in some conditions (different data points and noise levels). The results show that although the application of information criteria prevents over-fitting and under-fitting in most cases, there are cases where we cannot avoid even involving many data points and low noise levels in ideal situations. The results also show that the distribution of the fitting errors is not always normally distributed, although the observational noise is Gaussian, which contradicts an assumption of the information criteria.

[1]  Rohan A. Baxter,et al.  MML and Bayesianism: similarities and differences: introduction to minimum encoding inference Part , 1994 .

[2]  A. Mees,et al.  On selecting models for nonlinear time series , 1995 .

[3]  Alistair Mees.  PARSIMONIOUS DYNAMICAL RECONSTRUCTION , 1993 .

[4]  A. Lanterman.  Schwarz, Wallace, and Rissanen: Intertwining Themes in Theories of Model Order Estimation , 1999 .

[5]  Kevin Judd,et al.  Modeling chaotic motions of a string from experimental data , 1996 .

[6]  C. S. Wallace,et al.  Estimation and Inference by Compact Coding , 1987 .

[7]  Michael Small,et al.  Comparisons of new nonlinear modeling techniques with applications to infant respiration , 1998 .

[8]  Howell Tong,et al.  Non-Linear Time Series , 1990 .

[9]  Kevin Judd,et al.  Building Optimal Models of Time Series , 2003 .

[10]  A. Lanterman.  Schwarz, Wallace, and Rissanen: Intertwining Themes in Theories of Model Selection , 2001 .

[11]  Jorma Rissanen,et al.  MDL Denoising , 2000, IEEE Trans. Inf. Theory.

[12]  Kevin Judd,et al.  A Comparative Study of Model Selection Methods for Nonlinear Time Series , 2004, Int. J. Bifurc. Chaos.

[13]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[14]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[15]  D. Kilminster.  Modelling dynamical systems via behaviour criteria , 2002 .

[16]  Tomomichi Nakamura.  Modelling nonlinear time series using selection methods and information criteria , 2003 .

[17]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[18]  Jonathan J. Oliver.  Introduction to Minimum Encoding Inference , 1994 .

[19]  Jorma Rissanen,et al.  The Minimum Description Length Principle in Coding and Modeling , 1998, IEEE Trans. Inf. Theory.

[20]  Kevin Judd,et al.  Embedding as a modeling problem , 1998 .

[21]  Kevin Judd,et al.  Refinements to Model Selection for Nonlinear Time Series , 2003, Int. J. Bifurc. Chaos.

[22]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[23]  Patrick E. McSharry,et al.  Better Nonlinear Models from Noisy Data: Attractors with Maximum Likelihood , 1999, chao-dyn/9912002.

[24]  H. Akaike.  A new look at the statistical model identification , 1974 .

[25]  Jonathan J. Oliver,et al.  MDL and MML: Similarities and differences , 1994 .

[26]  Huaiyu Zhu.  On Information and Sufficiency , 1997 .

[27]  Michael Small,et al.  Modeling Nonlinear Time Series Using Improved Least Squares Method , 2006, Int. J. Bifurc. Chaos.

[28]  M. Small,et al.  Detecting periodicity in experimental data using linear modeling techniques , 1998, physics/9810021.

[29]  M. Hénon,et al.  A two-dimensional mapping with a strange attractor , 1976 .

[30]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .

[31]  Robert Graham,et al.  Quantization of a two-dimensional map with a strange attractor , 1983 .

[32]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[33]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[34]  Farmer,et al.  Predicting chaotic time series. , 1987, Physical review letters.