Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model

Numerous ways to meta-analyze single-case data have been proposed in the literature; however, consensus has not been reached on the most appropriate method. One method that has been proposed involves multilevel modeling. For this study, we used Monte Carlo methods to examine the appropriateness of Van den Noortgate and Onghena’s (2008) raw-data multilevel modeling approach for the meta-analysis of single-case data. Specifically, we examined the fixed effects (e.g., the overall average treatment effect) and the variance components (e.g., the between-person within-study variance in the treatment effect) in a three-level multilevel model (repeated observations nested within individuals, nested within studies). More specifically, bias of the point estimates, confidence interval coverage rates, and interval widths were examined as a function of the number of primary studies per meta-analysis, the modal number of participants per primary study, the modal series length per primary study, the level of autocorrelation, and the variances of the error terms. The degree to which the findings of this study are supportive of using Van den Noortgate and Onghena’s (2008) raw-data multilevel modeling approach to meta-analyzing single-case data depends on the particular parameter of interest. Estimates of the average treatment effect tended to be unbiased and produced confidence intervals that tended to overcover, but did come close to the nominal level as Level-3 sample size increased. Conversely, estimates of the variance in the treatment effect tended to be biased, and the confidence intervals for those estimates were inaccurate.

[1]  Thomas E. Scruggs,et al.  The Quantitative Synthesis of Single-Subject Research , 1987 .

[2]  Gilbert W. Fellingham,et al.  Performance of the Kenward–Roger Method when the Covariance Structure is Selected Using AIC and BIC , 2005 .

[3]  Patrick Onghena,et al.  A multilevel meta-analysis of single-subject experimental design studies , 2008 .

[4]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[5]  W J Gingerich,et al.  Methodological observations on applied behavioral science. , 1984, The Journal of applied behavioral science.

[6]  Samuel B. Green,et al.  The Impact of Misspecifying the Within-Subject Covariance Structure in Multiwave Longitudinal Multilevel Models: A Monte Carlo Study , 2007 .

[7]  J. Levin,et al.  Single-case research design and analysis : new directions for psychology and education , 1992 .

[8]  Russell D. Wolfinger,et al.  The Analysis of Repeated Measurements with Mixed-Model Adjusted F Tests , 2004 .

[9]  Wim Van Den Noortgate,et al.  The Aggregation of Single-Case Results Using Hierarchical Linear Models. , 2007 .

[10]  Patrick Onghena,et al.  Customization of pain treatments: single-case design and analysis. , 2005, The Clinical journal of pain.

[11]  Ann Casey,et al.  A Methodology for the Quantitative Synthesis of Intra-Subject Design Research , 1985 .

[12]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[13]  Ronald C. Serlin,et al.  Meta-analysis for single-case research. , 1992 .

[14]  M. Kenward,et al.  Small sample inference for fixed effects from restricted maximum likelihood. , 1997, Biometrics.

[15]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[16]  G. B. Schaalje,et al.  Adequacy of approximations to distributions of test statistics in complex mixed linear models , 2002 .

[17]  Rachel T. Fouladi,et al.  A Comparison of Two General Approaches to Mixed Model Longitudinal Analyses Under Small Sample Size Conditions , 2004 .

[18]  P. Onghena,et al.  Combining single-case experimental data using hierarchical linear models. , 2003 .

[19]  John M Ferron,et al.  Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches , 2010, Behavior research methods.

[20]  Keenan A. Pituch,et al.  The Performance of Multilevel Growth Curve Models Under an Autoregressive Moving Average Process , 2009 .

[21]  Cora J. M. Maas,et al.  Robustness issues in multilevel regression analysis , 2004 .

[22]  R. N. Kackar,et al.  Approximations for Standard Errors of Estimators of Fixed and Random Effects in Mixed Linear Models , 1984 .

[23]  Susan T. Hibbard,et al.  Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches , 2009, Behavior research methods.

[24]  P. Onghena,et al.  Hierarchical linear models for the quantitative integration of effect sizes in single-case research , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[25]  Alan E. Kazdin,et al.  Single-Case Research Designs: Methods for Clinical and Applied Settings , 2010 .

[26]  B. Gorman,et al.  Calculating effect sizes for meta-analysis: the case of the single case. , 1993, Behaviour research and therapy.