The Kullback information criterion for mixture regression models

We consider the problem of jointly selecting the number of components and variables in finite mixture regression models. The classical model selection criterion, AIC or BIC, may not be satisfactory in this setting, especially when the sample size is small or the number of variables is large. Specifically, they fit too many components and retain too many variables. An alternative mixture regression criterion, called MRC, which simultaneously determines the number of components and variables in mixture regression models, was proposed by Naik et al. (2007). In the same setting, we propose a new information criterion, called , for the simultaneous determination of the number of components and predictors. is based on the Kullback symmetric divergence instead of the Kullback directed divergence used for MRC. We show that the new criterion performs well than MRC in a small simulation study.

[1]  Abd-Krim Seghouane,et al.  A small sample model selection criterion based on Kullback's symmetric divergence , 2004, IEEE Transactions on Signal Processing.

[2]  Solomon Kullback,et al.  Information Theory and Statistics , 1970, The Mathematical Gazette.

[3]  Walter Zucchini,et al.  Model Selection , 2011, International Encyclopedia of Statistical Science.

[4]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[5]  R. Quandt.  A New Approach to Estimating Switching Regressions , 1972 .

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[8]  Prasad A. Naik,et al.  Extending the Akaike Information Criterion to Mixture Regression Models , 2007 .

[9]  Stephen E. Fienberg,et al.  A Celebration of Statistics , 1985 .

[10]  Kert Viele,et al.  Modeling with Mixtures of Linear Regressions , 2002, Stat. Comput..

[11]  A. Mkhadri,et al.  An Akaike Criterion based on Kullback Symmetric Divergence in the Presence of Incomplete-Data , 2009 .

[12]  Gérard Govaert,et al.  Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[14]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[15]  Clifford M. Hurvich,et al.  Regression and time series model selection in small samples , 1989 .

[16]  J. Cavanaugh.  A large-sample model selection criterion based on Kullback's symmetric divergence , 1999 .

[17]  M. Vannucci,et al.  Bayesian Variable Selection in Clustering High-Dimensional Data , 2005 .

[18]  A. Raftery,et al.  Modeling flat stretches, bursts, and outliers in time series using mixture transition distribution models , 1996 .

[19]  Abdallah Mkhadri,et al.  A corrected Akaike criterion based on Kullback's symmetric divergence: applications in time series, multiple and multivariate regression , 2006, Comput. Stat. Data Anal..

[20]  W. Li,et al.  On a mixture autoregressive model , 2000 .

[21]  H. Akaike.  Prediction and Entropy , 1985 .