Learning Coefficient of Vandermonde Matrix-Type Singularities in Model Selection

In recent years, selecting appropriate learning models has become more important with the increased need to analyze learning systems, and many model selection methods have been developed. The learning coefficient in Bayesian estimation, which serves to measure the learning efficiency in singular learning models, has an important role in several information criteria. The learning coefficient in regular models is known as the dimension of the parameter space over two, while that in singular models is smaller and varies in learning models. The learning coefficient is known mathematically as the log canonical threshold. In this paper, we provide a new rational blowing-up method for obtaining these coefficients. In the application to Vandermonde matrix-type singularities, we show the efficiency of such methods.

[1]  Shaowei Lin,et al.  Asymptotic Approximation of Marginal Likelihood Integrals , 2010 .

[2]  Piotr Zwiernik.  An Asymptotic Behaviour of the Marginal Likelihood for General Markov Models , 2011, J. Mach. Learn. Res..

[3]  Miki Aoyagi,et al.  Consideration on Singularities in Learning Theory and the Learning Coefficient , 2013, Entropy.

[4]  Miki Aoyagi,et al.  Resolution of Singularities and the Generalization Error with Bayesian Estimation for Layered Neural Network , 2005 .

[5]  Sumio Watanabe,et al.  Asymptotic analysis of Bayesian generalization error with Newton diagram , 2010, Neural Networks.

[6]  Sumio Watanabe,et al.  Exchange Monte Carlo Sampling From Bayesian Posterior for Singular Learning Machines , 2008, IEEE Transactions on Neural Networks.

[7]  M. Plummer,et al.  A Bayesian information criterion for singular models , 2013, 1309.0911.

[8]  Miki Aoyagi,et al.  A Bayesian Learning Coefficient of Generalization Error and Vandermonde Matrix-Type Singularities , 2010 .

[9]  H. Hironaka.  Resolution of Singularities of an Algebraic Variety Over a Field of Characteristic Zero: II , 1964 .

[10]  渡邊 澄夫.  Algebraic geometry and statistical learning theory , 2009 .

[11]  Piotr Zwiernik,et al.  Marginal likelihood and model selection for Gaussian latent tree and forest models , 2014, 1412.8285.

[12]  Sumio Watanabe.  Mathematical Theory of Bayesian Statistics , 2018 .

[13]  Sumio Watanabe,et al.  Algebraic Analysis for Nonidentifiable Learning Machines , 2001, Neural Computation.

[14]  Dan Geiger,et al.  Asymptotic Model Selection for Naive Bayesian Networks , 2002, J. Mach. Learn. Res..

[15]  Miki Aoyagi,et al.  Log canonical threshold of Vandermonde matrix type singularities and generalization error of a three layered neural network in Bayesian estimation , 2012 .

[16]  Sumio Watanabe,et al.  Stochastic complexities of reduced rank regression in Bayesian estimation , 2005, Neural Networks.

[17]  Kenji Nagata,et al.  Learning Coefficient of Generalization Error in Bayesian Estimation and Vandermonde Matrix-Type Singularity , 2012, Neural Computation.

[18]  H. Akaike.  A new look at the statistical model identification , 1974 .

[19]  Sumio Watanabe,et al.  Asymptotic behavior of exchange ratio in exchange Monte Carlo method , 2008, Neural Networks.

[20]  Sumio Watanabe,et al.  Equations of States in Singular Statistical Estimation , 2007, Neural Networks.

[21]  Sumio Watanabe.  Algebraic geometrical methods for hierarchical learning machines , 2001, Neural Networks.

[22]  Miki Aoyagi,et al.  Learning Coefficient in Bayesian Estimation of Restricted Boltzmann Machine , 2013 .

[23]  Michael Atiyah,et al.  Resolution of Singularities and Division of Distributions , 1970 .