The consequences of modeling autocorrelation when synthesizing single-case studies using a three-level model

Results from single-case studies are being synthesized using three-level models in which repeated observations are nested in participants, which in turn are nested in studies. We examined the performance of these models under conditions in which the errors associated with the repeated observations (the Level-1 errors) were assumed to be first-order autoregressive. Monte Carlo methods were used to examine conditions in which the first-order autoregressive assumption was accurate, conditions in which it represented an overspecification because the errors were actually independent, and conditions in which it represented a misspecification because the errors were generated on the basis of a moving-average model. Conditions also varied the series lengths, the numbers of participants per study, the numbers of studies per meta-analysis, the variances between the participants within studies, and the variances between studies. Fixed effects (e.g., the average treatment effect for the intervention and the average treatment effect for the trend) tended to be unbiased, and confidence intervals for the fixed effects tended to be accurate even when the error covariance model was overspecified or misspecified. The variance components, particularly at Levels 2 and 3, showed substantial bias.

[1]  T Kratochwill,et al.  A further consideration in the application of an analysis-of-variance model for the intrasubject replication design. , 1974, Journal of applied behavior analysis.

[2]  Larry E. Toothaker,et al.  N = 1 Designs: The Failure of ANOVA-Based Tests , 1983 .

[3]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[4]  Thomas E. Scruggs,et al.  The Quantitative Synthesis of Single-Subject Research , 1987 .

[5]  Kenneth M. Greenwood,et al.  Problems with the application of interrupted time series analysis for brief single-subject data , 1990 .

[6]  R. Wolfinger.  Covariance structure selection in general mixed models , 1993 .

[7]  John M. Ferron,et al.  Analyzing Single-Case Data: The Power of Randomization Tests. , 1995 .

[8]  Patrick Onghena,et al.  The power of randomization tests for single-case phase designs , 1996 .

[9]  A. Boomsma,et al.  Robustness Studies in Covariance Structure Modeling , 1998 .

[10]  J. McKean,et al.  Design Specification Issues in Time-Series Intervention Models , 2000 .

[11]  John Ferron,et al.  Effects of Misspecifying the First-Level Error Structure in Two-Level Models of Change , 2002, Multivariate behavioral research.

[12]  P. Onghena,et al.  Combining single-case experimental data using hierarchical linear models. , 2003 .

[13]  John Ferron,et al.  Tests for the Visual Analysis of Response-Guided Multiple-Baseline Data , 2006 .

[14]  Kimberly J. Vannest,et al.  Percentage of All Non-Overlapping Data (PAND) , 2007 .

[15]  Samuel B. Green,et al.  The Impact of Misspecifying the Within-Subject Covariance Structure in Multiwave Longitudinal Multilevel Models: A Monte Carlo Study , 2007 .

[16]  Patrick Onghena,et al.  A multilevel meta-analysis of single-subject experimental design studies , 2008 .

[17]  Patrick Onghena,et al.  An R package for single-case randomization tests , 2008, Behavior research methods.

[18]  S. Natasha Beretvas,et al.  A review of meta-analyses of single-subject experimental designs: Methodological issues and practice , 2008 .

[19]  Patrick Onghena,et al.  Randomization tests for multiple-baseline designs: An extension of the SCRT-R package , 2009, Behavior research methods.

[20]  Susan T. Hibbard,et al.  Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches , 2009, Behavior research methods.

[21]  Alan E. Kazdin,et al.  Single-Case Research Designs: Methods for Clinical and Applied Settings , 2010 .

[22]  John M Ferron,et al.  Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches , 2010, Behavior research methods.

[23]  W. Shadish,et al.  Characteristics of single-case designs used to assess intervention effects in 2008 , 2011, Behavior research methods.

[24]  Kimberly J. Vannest,et al.  Combining nonoverlap and trend for single-case research: Tau-U. , 2011, Behavior therapy.

[25]  Kimberly J. Vannest,et al.  Effect Size in Single-Case Research: A Review of Nine Nonoverlap Techniques , 2011, Behavior modification.

[26]  Eugene S. Edgington,et al.  Randomization Tests , 2011, International Encyclopedia of Statistical Science.

[27]  John M Ferron,et al.  Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model , 2012, Behavior research methods.

[28]  W. Shadish,et al.  A standardized mean difference effect size for single case designs , 2012, Research synthesis methods.

[29]  John M Ferron,et al.  Nonparametric statistical tests for single-case systematic and randomized ABAB…AB and alternating treatment intervention designs: new developments, new directions. , 2012, Journal of school psychology.

[30]  James E. Pustejovsky,et al.  A standardized mean difference effect size for multiple baseline designs across individuals , 2013, Research synthesis methods.

[31]  John M Ferron,et al.  The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study , 2013, Multivariate behavioral research.

[32]  Eden Nagler Kyse,et al.  Analyzing data from single-case designs using multilevel models: new applications and some agenda items for future research. , 2013, Psychological methods.

[33]  Wim Van den Noortgate,et al.  Bias Corrections for Standardized Effect Size Estimates Used With Single-Subject Experimental Designs , 2014 .

[34]  Mariola Moeyaert.  Three-level synthesis of single-subject experimental data: Further extensions, empirical validation and applications , 2014 .

[35]  John M Ferron,et al.  The use of multilevel analysis for integrating single-case experimental design results within a study and across studies , 2014, Neuropsychological rehabilitation.

[36]  Jeffrey J Borckardt,et al.  Simulation modelling analysis for small sets of single-subject data collected over time , 2014, Neuropsychological rehabilitation.

[37]  Mariola Moeyaert,et al.  Three-Level Analysis of Single-Case Experimental Data: Empirical Validation , 2014 .

[38]  Hariharan Swaminathan,et al.  An effect size measure and Bayesian analysis of single-case designs. , 2014, Journal of school psychology.

[39]  John M Ferron,et al.  Estimating causal effects from multiple-baseline studies: implications for design and analysis. , 2014, Psychological methods.

[40]  Patrick Onghena,et al.  Analysis of single-case data: Randomisation tests for measures of effect size , 2014, Neuropsychological rehabilitation.

[41]  Merlande Petit-Bois.  A Monte Carlo Study: The Consequences of the Misspecification of the Level-1 Error Structure when Meta-Analyzing Single-Case Designs , 2014 .

[42]  John M Ferron,et al.  From a single-level analysis to a multilevel analysis of single-case experimental designs. , 2014, Journal of school psychology.