WAIC AND WBIC ARE INFORMATION CRITERIA FOR SINGULAR STATISTICAL MODEL EVALUATION

Many statistical models and learning machines which have hierarchical structures, hidden variables, and grammatical rules are not regular but singular statistical models. In singular models, the log likelihood function can not be approximated by any quadratic form of a parameter, resulting that conventional information criteria such as AIC, BIC, TIC or DIC can not be used for model evaluation. Recently, new information criteria, WAIC and WBIC, were proposed based on singular learning theory. They can be applicable even if a true distribution is singular for or unrealizable by a statistical model. In this paper, we introduce definitions of WAIC and WBIC and discuss their fundamental properties.

[1]  Kenji Nagata,et al.  Learning Coefficient of Generalization Error in Bayesian Estimation and Vandermonde Matrix-Type Singularity , 2012, Neural Computation.

[2]  Sumio Watanabe,et al.  Stochastic complexities of reduced rank regression in Bayesian estimation , 2005, Neural Networks.

[3]  Sumio Watanabe,et al.  Singularities in complete bipartite graph-type Boltzmann machines and upper bounds of stochastic complexities , 2005, IEEE Transactions on Neural Networks.

[4]  Sumio Watanabe,et al.  A widely applicable Bayesian information criterion , 2012, J. Mach. Learn. Res..

[5]  Sumio Watanabe,et al.  Algebraic Analysis for Nonidentifiable Learning Machines , 2001, Neural Computation.

[6]  Piotr Zwiernik,et al.  Asymptotic model selection and identifiability of directed tree models with hidden variables , 2010 .

[7]  Sumio Watanabe,et al.  Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory , 2010, J. Mach. Learn. Res..

[8]  Kazuho Watanabe,et al.  Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation , 2006, J. Mach. Learn. Res..

[9]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[10]  Shinichi Nakajima,et al.  Generalization Performance of Subspace Bayes Approach in Linear Neural Networks , 2006, IEICE Trans. Inf. Syst..

[11]  渡邊 澄夫.  Algebraic geometry and statistical learning theory , 2009 .

[12]  Sumio Watanabe,et al.  Equations of States in Statistical Learning for an Unrealizable and Regular Case , 2010, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[13]  Masato Okada,et al.  Bayesian spectral deconvolution with the exchange Monte Carlo method , 2012, Neural Networks.

[14]  Dan Geiger,et al.  Asymptotic Model Selection for Naive Bayesian Networks , 2002, J. Mach. Learn. Res..

[15]  Sumio Watanabe,et al.  Information criterion for variational Bayes learning in regular and singular cases , 2012, The 6th International Conference on Soft Computing and Intelligent Systems, and The 13th International Symposium on Advanced Intelligence Systems.

[16]  Sumio Watanabe,et al.  Asymptotic behavior of exchange ratio in exchange Monte Carlo method , 2008, Neural Networks.

[17]  Sumio Watanabe,et al.  Singularities in mixture models and upper bounds of stochastic complexity , 2003, Neural Networks.

[18]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[19]  H. Akaike.  A new look at the statistical model identification , 1974 .

[20]  Aki Vehtari,et al.  A survey of Bayesian predictive methods for model assessment, selection and comparison , 2012 .

[21]  Sumio Watanabe.  Algebraic geometrical methods for hierarchical learning machines , 2001, Neural Networks.

[22]  Sumio Watanabe,et al.  Equations of States in Singular Statistical Estimation , 2007, Neural Networks.