Lack of confidence in approximate Bayesian computation model choice

Approximate Bayesian computation (ABC) have become an essential tool for the analysis of complex stochastic models. Grelaud et al. [(2009) Bayesian Anal 3:427–442] advocated the use of ABC for model choice in the specific case of Gibbs random fields, relying on an intermodel sufficiency property to show that the approximation was legitimate. We implemented ABC model choice in a wide range of phylogenetic models in the Do It Yourself-ABC (DIY-ABC) software [Cornuet et al. (2008) Bioinformatics 24:2713–2719]. We now present arguments as to why the theoretical arguments for ABC model choice are missing, because the algorithm involves an unknown loss of information induced by the use of insufficient summary statistics. The approximation error of the posterior probabilities of the models under comparison may thus be unrelated with the computational effort spent in running an ABC algorithm. We then conclude that additional empirical verifications of the performances of the ABC procedure as those available in DIY-ABC are necessary to conduct model choice.

[1]  Jean-Marie Cornuet,et al.  Bridgehead Effect in the Worldwide Invasion of the Biocontrol Harlequin Ladybird , 2010, PloS one.

[2]  Jean-Marie Hombert,et al.  Origins and Genetic Diversity of Pygmy Hunter-Gatherers from Western Central Africa , 2009, Current Biology.

[3]  Alan R Templeton,et al.  Coherent and incoherent inference in phylogeography and human evolution , 2010, Proceedings of the National Academy of Sciences.

[4]  Kevin R. Thornton,et al.  Experiments with the Site Frequency Spectrum , 2011, Bulletin of mathematical biology.

[5]  Richard G. Everitt,et al.  Likelihood-free estimation of model evidence , 2011 .

[6]  Christophe Andrieu,et al.  Model criticism based on likelihood-free inference, with an application to protein network evolution , 2009, Proceedings of the National Academy of Sciences.

[7]  Daniel Wegmann,et al.  Bayesian Computation and Model Selection Without Likelihoods , 2010, Genetics.

[8]  Nicholas G. Polson,et al.  Data augmentation for support vector machines , 2011 .

[9]  Olivier François,et al.  Invalid arguments against ABC: Reply to A.R. Templeton , 2010 .

[10]  Laurent Excoffier,et al.  Bayesian inference of the demographic history of chimpanzees. , 2010, Molecular biology and evolution.

[11]  R. Wilkinson Approximate Bayesian computation (ABC) gives exact results under the assumption of model error , 2008, Statistical applications in genetics and molecular biology.

[12]  L. Excoffier,et al.  Statistical evaluation of alternative models of human evolution , 2007, Proceedings of the National Academy of Sciences.

[13]  Luping Zhao,et al.  A Bayesian Semiparametric Temporally-Stratified Proportional Hazards Model with Spatial Frailties. , 2012, Bayesian analysis.

[14]  Paul Fearnhead,et al.  Constructing summary statistics for approximate Bayesian computation: semi‐automatic approximate Bayesian computation , 2012 .

[15]  L. M. M.-T. Theory of Probability , 1929, Nature.

[16]  Laurent Excoffier,et al.  ABCtoolbox: a versatile toolkit for approximate Bayesian computations , 2010, BMC Bioinformatics.

[17]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[18]  M. Beaumont,et al.  ABC: a useful Bayesian tool for the analysis of population data. , 2010, Infection, genetics and evolution : journal of molecular epidemiology and evolutionary genetics in infectious diseases.

[19]  Christian P Robert,et al.  Incoherent phylogeographic inference , 2010, Proceedings of the National Academy of Sciences.

[20]  Arnaud Estoup,et al.  Multiple Transatlantic Introductions of the Western Corn Rootworm , 2005, Science.

[21]  P. Donnelly,et al.  Inferring coalescence times from DNA sequence data. , 1997, Genetics.

[22]  G Barbujani,et al.  Comparing models on the genealogical relationships among Neandertal, Cro-Magnoid and modern Europeans by serial coalescent simulations , 2009, Heredity.

[23]  David Welch,et al.  Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems , 2009, Journal of The Royal Society Interface.

[24]  Jean-Michel Marin,et al.  Approximate Bayesian computational methods , 2011, Statistics and Computing.

[25]  Gareth W. Peters,et al.  Parameter Estimation for Hidden Markov Models with Intractable Likelihoods , 2011, 1103.5399.

[26]  Michael P. H. Stumpf,et al.  Simulation-based model selection for dynamical systems in systems and population biology , 2009, Bioinform..

[27]  Mark A. Beaumont,et al.  Approximate Bayesian Computation Without Summary Statistics: The Case of Admixture , 2009, Genetics.

[28]  Jean-Marie Cornuet,et al.  Inferring population history with DIY ABC: a user-friendly approach to approximate Bayesian computation , 2008, Bioinform..

[29]  Jukka Corander,et al.  In defence of model‐based inference in phylogeography , 2010, Molecular ecology.

[30]  A. Estoup,et al.  Bayesian inferences on the recent island colonization history by the bird Zosterops lateralis lateralis , 2003, Molecular ecology.

[31]  Gil B. Carvalho,et al.  Reply to Piper et al.: Drosophila dietary restriction—Does it hold water? , 2010, Proceedings of the National Academy of Sciences of the United States of America.

[32]  Christoph Leuenberger Daniel Wegmann Laurent Excoffier Bayesian Computation and Model Selection in Population Genetics , 2009, 0901.2231.

[33]  Jean-Marie Hombert,et al.  Inferring the Demographic History of African Farmers and Pygmy Hunter–Gatherers Using a Multilocus Resequencing Data Set , 2009, PLoS genetics.

[34]  R. Huey,et al.  Introduction history of Drosophila subobscura in the New World: a microsatellite‐based survey using ABC methods , 2007, Molecular ecology.

[35]  C. Robert,et al.  ABC likelihood-free methods for model choice in Gibbs random fields , 2008, 0807.2767.

[36]  M. Feldman,et al.  Population growth of human Y chromosomes: a study of Y chromosome microsatellites. , 1999, Molecular biology and evolution.

[37]  Guido Barbujani,et al.  Inferring genealogical processes from patterns of Bronze-Age and modern DNA variation in Sardinia. , 2010, Molecular biology and evolution.

[38]  Jean-Marie Cornuet,et al.  GENETIC ANALYSIS OF COMPLEX DEMOGRAPHIC SCENARIOS: SPATIALLY EXPANDING POPULATIONS OF THE CANE TOAD, BUFO MARINUS , 2004, Evolution; international journal of organic evolution.

[39]  Jean-Marie Cornuet,et al.  Inference on population history and model checking using DNA sequence and microsatellite data with the software DIYABC (v1.0) , 2010, BMC Bioinformatics.

[40]  O. François,et al.  Approximate Bayesian Computation (ABC) in practice. , 2010, Trends in ecology & evolution.

[41]  Alan R Templeton,et al.  Statistical hypothesis testing in intraspecific phylogeography: nested clade phylogeographical analysis vs. approximate Bayesian computation , 2009, Molecular ecology.

[42]  D. Balding,et al.  Statistical Applications in Genetics and Molecular Biology On Optimal Selection of Summary Statistics for Approximate Bayesian Computation , 2011 .

[43]  M. Beaumont,et al.  Inferring introduction routes of invasive species using approximate Bayesian computation on microsatellite data , 2010, Heredity.

[44]  Paul Fearnhead,et al.  Semi-automatic Approximate Bayesian Computation , 2010 .

[45]  R. Plevin,et al.  Approximate Bayesian Computation in Evolution and Ecology , 2011 .

[46]  U. Ramakrishnan,et al.  Using phylochronology to reveal cryptic population histories: review and synthesis of 29 ancient DNA studies , 2009, Molecular ecology.

[47]  Paul Marjoram,et al.  Statistical Applications in Genetics and Molecular Biology Approximately Sufficient Statistics and Bayesian Computation , 2011 .

[48]  H. Savenije,et al.  The water footprint of bioenergy from Jatropha curcas L. , 2009, Proceedings of the National Academy of Sciences.

[49]  D. Balding,et al.  Approximate Bayesian computation in population genetics. , 2002, Genetics.

[50]  Joao S. Lopes,et al.  PopABC: a program to infer historical demographic parameters , 2009, Bioinform..