Multilevel meta-analysis of single-subject experimental designs: A simulation study

One way to combine data from single-subject experimental design studies is by performing a multilevel meta-analysis, with unstandardized or standardized regression coefficients as the effect size metrics. This study evaluates the performance of this approach. The results indicate that a multilevel meta-analysis of unstandardized effect sizes results in good estimates of the effect. The multilevel meta-analysis of standardized effect sizes, on the other hand, is suitable only when the number of measurement occasions for each subject is 20 or more. The effect of the treatment on the intercept is estimated with enough power when the studies are homogeneous or when the number of studies is large; the power of the effect on the slope is estimated with enough power only when the number of studies and the number of measurement occasions are large.

[1]  Ann Casey,et al.  A Methodology for the Quantitative Synthesis of Intra-Subject Design Research , 1985 .

[2]  van der Eddy Zee,et al.  Proceedings of Measuring Behavior 2014 , 2014 .

[3]  Ke-Hai Yuan,et al.  Biases and Standard Errors of Standardized Regression Coefficients , 2011, Psychometrika.

[4]  Erin E. Barton,et al.  Comparison of Overlap Methods for Quantitatively Synthesizing Single-Subject Data , 2010 .

[5]  S. Natasha Beretvas,et al.  A review of meta-analyses of single-subject experimental designs: Methodological issues and practice , 2008 .

[6]  W J Gingerich,et al.  Methodological observations on applied behavioral science. , 1984, The Journal of applied behavioral science.

[7]  L. Hedges.  Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[8]  R. Parrila,et al.  Examining the effectiveness of peer-mediated and video-modeling social skills interventions for children with autism spectrum disorders: A meta-analysis in single-case research using HLM , 2011 .

[9]  R. Littell.  SAS System for Mixed Models , 1996 .

[10]  Patrick Onghena,et al.  A multilevel meta-analysis of single-subject experimental design studies , 2008 .

[11]  L. Kern,et al.  Social Story™ Interventions for Students with Autism Spectrum Disorders: A Meta-Analysis , 2010, Journal of autism and developmental disorders.

[12]  D. Barlow,et al.  Single Case Experimental Designs: Strategies for Studying Behavior Change , 1976 .

[13]  B. Gorman,et al.  Calculating effect sizes for meta-analysis: the case of the single case. , 1993, Behaviour research and therapy.

[14]  Thomas E. Scruggs,et al.  The Quantitative Synthesis of Single-Subject Research , 1987 .

[15]  Susan T. Hibbard,et al.  Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches , 2009, Behavior research methods.

[16]  Kelvyn Jones.  Review of HLM 4 for Windows , 1996 .

[17]  Kimberly J. Vannest,et al.  Effect Size in Single-Case Research: A Review of Nine Nonoverlap Techniques , 2011, Behavior modification.

[18]  C. Sachse-Lee,et al.  A Meta-Analysis of Single-Subject-Design Intervention Research for Students with LD , 2000, Journal of learning disabilities.

[19]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[20]  W. Shadish,et al.  Characteristics of single-case designs used to assess intervention effects in 2008 , 2011, Behavior research methods.

[21]  S. Raudenbush,et al.  A multivariate mixed linear model for meta-analysis. , 1996 .

[22]  Karrie A. Shogren,et al.  The Effect of Choice-Making as an Intervention for Problem Behavior , 2004 .

[23]  P. Onghena,et al.  Hierarchical linear models for the quantitative integration of effect sizes in single-case research , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[24]  P. Onghena,et al.  Combining single-case experimental data using hierarchical linear models. , 2003 .

[25]  Wim Van den Noortgate,et al.  Self-injurious behavior in people with profound intellectual disabilities: a meta-analysis of single-case studies. , 2011, Research in developmental disabilities.

[26]  S. West,et al.  Statistical issues in the study of temporal data: daily experiences. , 1991, Journal of personality.

[27]  Alan E. Kazdin,et al.  On resolving ambiguities of the multiple-baseline design: Problems and recommendations , 1975 .

[28]  Hariharan Swaminathan,et al.  A generalized least squares regression approach for computing effect sizes in single-case research: application examples. , 2011, Journal of school psychology.

[29]  Daniel M. Maggin,et al.  A Quantitative Synthesis of Methodology in the Meta-Analysis of Single-Subject Research for Students with Disabilities: 1985–2009 , 2011 .

[30]  Jeffrey C. Valentine,et al.  Research Synthesis and Meta-Analysis , 2008 .

[31]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[32]  Harvey Goldstein.  Consistent estimators for multilevel generalised linear models using an iterated bootstrap , 1998 .

[33]  M. Heyvaert,et al.  A multilevel meta-analysis of single-case and small-n research on interventions for reducing challenging behavior in persons with intellectual disabilities. , 2012, Research in developmental disabilities.

[34]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[35]  John M Ferron,et al.  Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches , 2010, Behavior research methods.

[36]  Bruce A. Thyer,et al.  Single-case experimental designs: Strategies for studying behavior change (2nd edn): by David H. Barlow and Michel Hersen Pergamon Press, New York, 1984, 419 pp. , 1985 .

[37]  Jonathan M. Campbell.  Statistical Comparison of Four Effect Sizes for Single-Subject Designs , 2004, Behavior modification.