Limitations of “Limitations of Bayesian Leave-one-out Cross-Validation for Model Selection”

In an earlier article in this journal, Gronau and Wagenmakers (2018) discuss some problems with leave-one-out cross-validation (LOO) for Bayesian model selection. However, the variant of LOO that Gronau and Wagenmakers discuss is at odds with a long literature on how to use LOO well. In this discussion, we discuss the use of LOO in practical data analysis, from the perspective that we need to abandon the idea that there is a device that will produce a single-number decision rule.

[1]  Carsten F. Dormann,et al.  Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure , 2017 .

[2]  Fulvio Spezzaferri,et al.  A Predictive Model Selection Criterion , 1984 .

[3]  Yoshua Bengio,et al.  No Unbiased Estimator of the Variance of K-Fold Cross-Validation , 2003, J. Mach. Learn. Res..

[4]  A. Gelman,et al.  Using Stacking to Average Bayesian Predictive Distributions (with Discussion) , 2017, Bayesian Analysis.

[5]  Eric-Jan Wagenmakers,et al.  Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection , 2018, Computational brain & behavior.

[6]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[7]  Aki Vehtari,et al.  Understanding predictive information criteria for Bayesian models , 2013, Statistics and Computing.

[8]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[9]  Aki Vehtari,et al.  A survey of Bayesian predictive methods for model assessment, selection and comparison , 2012 .

[10]  Aki Vehtari,et al.  Projective Inference in High-dimensional Problems: Prediction and Feature Selection , 2018, Electronic Journal of Statistics.

[11]  Marco A. R. Ferreira,et al.  Spatiotemporal Models for Gaussian Areal Data , 2009 .

[12]  Thiago G. Martins,et al.  Penalising Model Component Complexity: A Principled, Practical Approach to Constructing Priors , 2014, 1403.4630.

[13]  Aki Vehtari,et al.  Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC , 2015, Statistics and Computing.

[14]  Aki Vehtari,et al.  Visualization in Bayesian workflow , 2017, Journal of the Royal Statistical Society: Series A (Statistics in Society).

[15]  Christian P. Robert,et al.  Variable selection in qualitative models via an entropic explanatory power , 2003 .

[16]  Aki Vehtari,et al.  Comparison of Bayesian predictive methods for model selection , 2015, Stat. Comput..

[17]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[18]  Aki Vehtari,et al.  Approximate leave-future-out cross-validation for time series models , 2019 .

[19]  Andrew Gelman,et al.  Diculty of selecting among multilevel models using predictive accuracy , 2015 .

[20]  C. Robert,et al.  Model choice in generalised linear models: A Bayesian approach via Kullback-Leibler projections , 1998 .