On Using Bayesian Methods to Address Small Sample Problems

As Bayesian methods continue to grow in accessibility and popularity, more empirical studies are turning to Bayesian methods to model small sample data. Bayesian methods do not rely on asympotics, a property that can be a hindrance when employing frequentist methods in small sample contexts. Although Bayesian methods are better equipped to model data with small sample sizes, estimates are highly sensitive to the specification of the prior distribution. If this aspect is not heeded, Bayesian estimates can actually be worse than frequentist methods, especially if frequentist small sample corrections are utilized. We show with illustrative simulations and applied examples that relying on software defaults or diffuse priors with small samples can yield more biased estimates than frequentist methods. We discuss conditions that need to be met if researchers want to responsibly harness the advantages that Bayesian methods offer for small sample problems as well as leading small sample frequentist methods.

[1]  M. Kenward,et al.  Small sample inference for fixed effects from restricted maximum likelihood. , 1997, Biometrics.

[2]  Sarah Depaoli,et al.  Iteration of Partially Specified Target Matrices: Applications in Exploratory and Bayesian Confirmatory Factor Analysis , 2015, Multivariate behavioral research.

[3]  R. Potthoff,et al.  A generalized multivariate analysis of variance model useful especially for growth curve problems , 1964 .

[4]  Larry R. Price,et al.  Small Sample Properties of Bayesian Multivariate Autoregressive Time Series Models , 2012 .

[5]  Scott A Baldwin,et al.  Bayesian methods for the analysis of small sample multilevel data with a complex variance structure. , 2013, Psychological methods.

[6]  Andrew Gelman,et al.  Why We (Usually) Don't Have to Worry About Multiple Comparisons , 2009, 0907.2478.

[7]  J. Kosfelder,et al.  Dialectical behavior therapy for borderline personality disorder: a meta-analysis using mixed-effects modeling. , 2010, Journal of consulting and clinical psychology.

[8]  Sarah Depaoli,et al.  The Impact of Inaccurate “Informative” Priors for Growth Parameters in Bayesian Growth Mixture Modeling , 2014 .

[9]  Bengt Muthén,et al.  Bayesian structural equation modeling: a more flexible representation of substantive theory. , 2012, Psychological methods.

[10]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[11]  Kevin J. Grimm,et al.  Comparison of Inverse Wishart and Separation-Strategy Priors for Bayesian Estimation of Covariance Parameter Matrix in Growth Curve Analysis , 2016 .

[12]  Laura M. Stapleton,et al.  Modeling Clustered Data with Very Few Clusters , 2016, Multivariate behavioral research.

[13]  B. Skinner,et al.  Dialectical behavior therapy. , 2002, The Harvard mental health letter.

[14]  G. Cumming.  The New Statistics: Why and How , 2013 .

[15]  Daniel Stegmueller,et al.  How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches , 2013 .

[16]  S. Rimm-Kaufman,et al.  Engagement in Training as a Mechanism to Understanding Fidelity of Implementation of the Responsive Classroom Approach , 2014, Prevention Science.

[17]  B. Muthén.  Bayesian Analysis In Mplus : A Brief Introduction , 2010 .

[18]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[19]  Daniel McNeish,et al.  Using Data-Dependent Priors to Mitigate Small Sample Bias in Latent Growth Models , 2016 .

[20]  Y. MacNab,et al.  Idealism and Relativism Across Cultures , 2011 .

[21]  A. Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[22]  Daniel J Bauer,et al.  Building path diagrams for multilevel models. , 2007, Psychological methods.

[23]  D. Rubin,et al.  Inference from Iterative Simulation Using Multiple Sequences , 1992 .

[24]  Sarah Depaoli,et al.  A Bayesian Approach to Multilevel Structural Equation Modeling With Continuous and Dichotomous Outcomes , 2015 .

[25]  Patrick J Curran,et al.  Have Multilevel Models Been Structural Equation Models All Along? , 2003, Multivariate behavioral research.

[26]  Daniel J Bauer,et al.  Distributional assumptions of growth mixture models: implications for overextraction of latent trajectory classes. , 2003, Psychological methods.

[27]  Michael G Kenward,et al.  The analysis of very small samples of repeated measurements I: An adjusted sandwich estimator , 2010, Statistics in medicine.

[28]  Tihomir Asparouhov,et al.  Bayesian Analysis of Latent Variable Models using Mplus , 2010 .

[29]  A. Boomsma,et al.  Robustness Studies in Covariance Structure Modeling , 1998 .

[30]  G. A. Marcoulides,et al.  Multilevel Analysis Techniques and Applications , 2002 .

[31]  Joseph B. Kadane,et al.  Bayesian Methods for Prevention Research , 2015, Prevention Science.

[32]  Sarah Depaoli,et al.  Improving Transparency and Replication in Bayesian Statistics: The WAMBS-Checklist , 2017, Psychological methods.

[33]  Franz J. Neyer,et al.  A Gentle Introduction to Bayesian Analysis: Applications to Developmental Research , 2013, Child development.

[34]  Michael G Kenward,et al.  The analysis of very small samples of repeated measurements II: A modified Box correction , 2010, Statistics in medicine.

[35]  Deborah Ashby,et al.  Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care , 2004, Clinical trials.

[36]  William J. Browne,et al.  Bayesian and likelihood-based methods in multilevel modeling 1 A comparison of Bayesian and likelihood-based methods for fitting multilevel models , 2006 .

[37]  M. Bartlett.  TESTS OF SIGNIFICANCE IN FACTOR ANALYSIS , 1950 .

[38]  K. Yuan,et al.  Structural Equation Modeling with Small Samples: Test Statistics. , 1999, Multivariate behavioral research.

[39]  P. Gaudreau,et al.  A point-by-point analysis of performance in a fencing match: psychological processes associated with winning and losing streaks. , 2014, Journal of sport & exercise psychology.

[40]  D. Russell.  In Search of Underlying Dimensions: The Use (and Abuse) of Factor Analysis in Personality and Social Psychology Bulletin , 2002 .

[41]  R. Levy.  The Rise of Markov Chain Monte Carlo Estimation for Psychometric Modeling , 2009 .

[42]  Michael G. Kenward,et al.  An improved approximation to the precision of fixed effects from restricted maximum likelihood , 2009, Comput. Stat. Data Anal..

[43]  David B. Dunson,et al.  Bayesian Structural Equation Modeling , 2007 .

[44]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[45]  J. Kruschke.  Doing Bayesian Data Analysis: A Tutorial with R and BUGS , 2010 .

[46]  R. Scheines,et al.  Bayesian estimation and testing of structural equation models , 1999 .

[47]  H. Goldstein,et al.  Meta‐analysis using multilevel models with an application to the study of class size effects , 2000 .

[48]  K. Yuan.  Fit Indices Versus Test Statistics , 2005, Multivariate behavioral research.

[49]  Laura M. Stapleton,et al.  The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration , 2014, Educational Psychology Review.

[50]  D. Dunson,et al.  Bayesian latent variable models for clustered mixed outcomes , 2000 .

[51]  Joop J. Hox,et al.  How few countries will do? Comparative survey analysis from a Bayesian perspective , 2012 .

[52]  Sarah Depaoli,et al.  Mixture class recovery in GMM under varying degrees of class separation: frequentist versus Bayesian estimation. , 2013, Psychological methods.

[53]  F. B. Gonçalves,et al.  An Integrated Bayesian Model for DIF Analysis , 2009 .

[54]  John R. Nesselroade,et al.  Bayesian analysis of longitudinal data using growth curve models , 2007 .

[55]  D B Dunson,et al.  Commentary: practical advantages of Bayesian analysis of epidemiologic data. , 2001, American journal of epidemiology.

[56]  Nicholas D. Myers,et al.  A Review of Meta-Analyses in Education , 2012 .

[57]  A. Stenling,et al.  Bayesian structural equation modeling in sport and exercise psychology. , 2015, Journal of sport & exercise psychology.

[58]  Ke-Hai Yuan,et al.  Mean and Covariance Structure Analysis: Theoretical and Practical Improvements , 1997 .

[59]  R. MacCallum,et al.  Applications of structural equation modeling in psychological research. , 2000, Annual review of psychology.

[60]  Patrick Onghena,et al.  Multilevel Meta-Analysis: A Comparison with Traditional Meta-Analytical Procedures , 2003 .

[61]  R. van de Schoot,et al.  Analyzing small data sets using Bayesian estimation: the case of posttraumatic stress symptoms following mechanical ventilation in burn survivors , 2015, European journal of psychotraumatology.

[62]  James G. Scott,et al.  On the half-cauchy prior for a global scale parameter , 2011, 1104.4937.

[63]  J. Schoeneberger.  The Impact of Sample Size and Other Factors When Estimating Multilevel Logistic Models , 2016 .

[64]  Ke-Hai Yuan,et al.  F Tests for Mean and Covariance Structure Analysis , 1999 .

[65]  S. D. Winter,et al.  A Systematic Review of Bayesian Articles in Psychology: The Last 25 Years , 2017, Psychological methods.

[66]  Craig K. Enders,et al.  Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. , 2007, Psychological methods.

[67]  M. Cheung.  A model for integrating fixed-, random-, and mixed-effects meta-analyses into structural equation modeling. , 2008, Psychological methods.

[68]  Simon Jackman,et al.  Bayesian Analysis for the Social Sciences , 2009 .

[69]  David R. Jones,et al.  How vague is vague? A simulation study of the impact of the use of vague prior distributions in MCMC using WinBUGS , 2005, Statistics in medicine.

[70]  Vincent S. Staggs,et al.  Comparison of naïve, Kenward–Roger, and parametric bootstrap interval approaches to small-sample inference in linear mixed models , 2017, Commun. Stat. Simul. Comput..

[71]  Xin-Yuan Song,et al.  Evaluation of the Bayesian and Maximum Likelihood Approaches in Analyzing Structural Equation Models with Small Sample Sizes , 2004, Multivariate behavioral research.

[72]  D. Harville.  Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems , 1977 .

[73]  Susan T. Hibbard,et al.  Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches , 2009, Behavior research methods.