A tutorial on Bayes factor estimation with the product space method

a b s t r a c t The Bayes factor is an intuitive and principled model selection tool from Bayesian statistics. The Bayes factor quantifies the relative likelihood of the observed data under two competing models, and as such, it measures the evidence that the data provides for one model versus the other. Unfortunately, computation of the Bayes factor often requires sampling-based procedures that are not trivial to implement. In this tutorial, we explain and illustrate the use of one such procedure, known as the product space method (Carlin & Chib, 1995). This is a transdimensional Markov chain Monte Carlo method requiring the construction of a ''supermodel'' encompassing the models under consideration. A model index measures the proportion of times that either model is visited to account for the observed data. This proportion can then be transformed to yield a Bayes factor. We discuss the theory behind the product space method and illustrate, by means of applied examples from psychological research, how the method can be implemented in practice.

[1]  Jun Lu,et al.  An introduction to Bayesian hierarchical models with an application in the theory of signal detection , 2005, Psychonomic bulletin & review.

[2]  L. Wasserman,et al.  Computing Bayes Factors by Combining Simulation and Asymptotic Approximations , 1997 .

[3]  Petros Dellaportas,et al.  On Bayesian model and variable selection using MCMC , 2002, Stat. Comput..

[4]  Frank Jäkel,et al.  Bayesian inference for psychometric functions. , 2005, Journal of vision.

[5]  J. Suls,et al.  Emotional Reactivity to Everyday Problems, Affective Inertia, and Neuroticism , 1998 .

[6]  R Ratcliff,et al.  Bias in the priming of object decisions. , 1995, Journal of experimental psychology. Learning, memory, and cognition.

[7]  A. Raftery,et al.  Estimating Bayes Factors via Posterior Simulation with the Laplace—Metropolis Estimator , 1997 .

[8]  Samuel D. Conte,et al.  Elementary Numerical Analysis: An Algorithmic Approach , 1975 .

[9]  Adrian F. M. Smith,et al.  Simple conditions for the convergence of the Gibbs sampler and Metropolis-Hastings algorithms , 1994 .

[10]  Joshua B. Tenenbaum,et al.  Combining causal and similarity-based reasoning , 2006, NIPS.

[11]  Michael D. Lee,et al.  A Hierarchical Bayesian Model of Human Decision-Making on an Optimal Stopping Problem , 2006, Cogn. Sci..

[12]  M. Mikulincer,et al.  The affective component of the secure base schema: affective priming with representations of attachment security. , 2001, Journal of personality and social psychology.

[13]  Michael D. Lee,et al.  The roles of the convex hull and number of intersections upon performance on visually presented traveling salesperson problems , 2002 .

[14]  H. Hoijtink.  Confirmatory Latent Class Analysis: Model Selection Using Bayes Factors and (Pseudo) Likelihood Ratio Statistics , 2001, Multivariate behavioral research.

[15]  Charles Kemp,et al.  The discovery of structural form , 2008, Proceedings of the National Academy of Sciences.

[16]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[17]  S. Godsill.  On the Relationship Between Markov chain Monte Carlo Methods for Model Uncertainty , 2001 .

[18]  P. Kuppens,et al.  Emotional Inertia and Psychological Maladjustment , 2010, Psychological science.

[19]  I. J. Myung,et al.  Applying Occam’s razor in modeling cognition: A Bayesian approach , 1997 .

[20]  Michael I. Jordan.  Graphical Models , 2003 .

[21]  Ben Taskar,et al.  Graphical Models in a Nutshell , 2007 .

[22]  M. T. Reinitz,et al.  Mechanisms of facilitation in primed perceptual identification , 1996, Memory & cognition.

[23]  C. Gallistel,et al.  The Importance of Proving the Null , 2022 .

[24]  M. Lee,et al.  Bayesian statistical inference in psychology: comment on Trafimow (2003). , 2005, Psychological review.

[25]  Jeffrey N. Rouder,et al.  A hierarchical process-dissociation model. , 2008, Journal of experimental psychology. General.

[26]  A. Brix.  Bayesian Data Analysis, 2nd edn , 2005 .

[27]  J B Poline,et al.  Cerebral mechanisms of word masking and unconscious repetition priming , 2001, Nature Neuroscience.

[28]  P. Green,et al.  On Bayesian Analysis of Mixtures with an Unknown Number of Components (with discussion) , 1997 .

[29]  N. Bolger,et al.  Diary methods: capturing life as it is lived. , 2003, Annual review of psychology.

[30]  Daniel L. Schacter,et al.  Priming and Multiple Memory Systems: Perceptual Mechanisms of Implicit Memory , 1992, Journal of Cognitive Neuroscience.

[31]  Jun Lu,et al.  Signal Detection Models with Random Participant and Item Effects , 2007 .

[32]  R Ratcliff,et al.  Bias effects in implicit memory tasks. , 1996, Journal of experimental psychology. General.

[33]  D B Boivin,et al.  Influence of sleep-wake and circadian rhythm disturbances in psychiatric disorders. , 2000, Journal of psychiatry & neuroscience : JPN.

[34]  D. Kahneman,et al.  A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method , 2004, Science.

[35]  Rupert Brown,et al.  Category and stereotype activation: Is prejudice inevitable? , 1997 .

[36]  P. Green.  Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[37]  M. Lee.  Three case studies in the Bayesian analysis of cognitive models , 2008, Psychonomic bulletin & review.

[38]  David J. Lunn,et al.  Generic reversible jump MCMC using graphical models , 2009, Stat. Comput..

[39]  E. Wagenmakers.  A practical solution to the pervasive problems ofp values , 2007, Psychonomic bulletin & review.

[40]  B. Carlin,et al.  Bayesian Model Choice Via Markov Chain Monte Carlo Methods , 1995 .

[41]  M. Lee.  How cognitive modeling can benefit from hierarchical Bayesian models. , 2011 .

[42]  I. J. Myung,et al.  Counting probability distributions: Differential geometry and model selection , 2000, Proc. Natl. Acad. Sci. USA.

[43]  M. Lee,et al.  A Bayesian analysis of human decision-making on bandit problems , 2009 .

[44]  René Zeelenberg,et al.  Priming in implicit memory tasks: prior study causes enhanced discriminability, not only bias. , 2002, Journal of experimental psychology. General.

[45]  Michael D. Lee,et al.  A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods , 2008, Cogn. Sci..

[46]  Ben Taskar,et al.  Introduction to statistical relational learning , 2007 .

[47]  M. Lee,et al.  The roles of the convex hull and the number of potential intersections in performance on visually presented traveling salesperson problems , 2003, Memory & cognition.

[48]  J. Geweke,et al.  Bayesian Inference in Econometric Models Using Monte Carlo Integration , 1989 .

[49]  L. Tierney,et al.  Accurate Approximations for Posterior Moments and Marginal Densities , 1986 .

[50]  J. Raaijmakers,et al.  How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test , 2009, Psychonomic bulletin & review.

[51]  Jeffrey N Rouder,et al.  Detecting chance: A solution to the null sensitivity problem in subliminal priming , 2007, Psychonomic bulletin & review.

[52]  A. Raftery.  Bayesian Model Selection in Social Research , 1995 .

[53]  E. Wagenmakers,et al.  Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method , 2010, Cognitive Psychology.

[54]  Emmanuel Dupoux,et al.  Subliminal Speech Priming , 2005, Psychological science.

[55]  Michael D. Lee.  Generating Additive Clustering Models with Minimal Stochastic Complexity , 2002, J. Classif..

[56]  C. Robert,et al.  Bayesian Modeling Using WinBUGS , 2009 .

[57]  Abraham P. Buunk,et al.  Judging a book by its cover: Jealousy after subliminal priming with attractive and unattractive faces , 2010 .

[58]  S. Dehaene,et al.  Imaging unconscious semantic priming , 1998, Nature.

[59]  Michael D. Lee,et al.  A Bayesian analysis of retention functions , 2004 .

[60]  Jeffrey N. Rouder,et al.  Bayesian t tests for accepting and rejecting the null hypothesis , 2009, Psychonomic bulletin & review.

[61]  Walter R. Gilks,et al.  A Language and Program for Complex Bayesian Modelling , 1994 .

[62]  Philippe Delespaul,et al.  Diurnal mood variation in major depressive disorder. , 2006, Emotion.

[63]  M. Lee,et al.  Statistical Evidence in Experimental Psychology , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[64]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[65]  S. Chib.  Marginal Likelihood from the Gibbs Output , 1995 .

[66]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[67]  Tapabrata Maiti,et al.  Bayesian Data Analysis (2nd ed.) (Book) , 2004 .

[68]  I. J. Myung,et al.  Toward a method of selecting among computational models of cognition. , 2002, Psychological review.