Analysis of single-case experimental count data using the linear mixed effects model: A simulation study

When (meta-)analyzing single-case experimental design (SCED) studies by means of hierarchical or multilevel modeling, applied researchers almost exclusively rely on the linear mixed model (LMM). This type of model assumes that the residuals are normally distributed. However, very often SCED studies consider outcomes of a discrete rather than a continuous nature, like counts, percentages or rates. In those cases the normality assumption does not hold. The LMM can be extended into a generalized linear mixed model (GLMM), which can account for the discrete nature of SCED count data. In this simulation study, we look at the effects of misspecifying an LMM for SCED count data simulated according to a GLMM. We compare the performance of a misspecified LMM and of a GLMM in terms of goodness of fit, fixed effect parameter recovery, type I error rate, and power. Because the LMM and the GLMM do not estimate identical fixed effects, we provide a transformation to compare the fixed effect parameter recovery. The results show that, compared to the GLMM, the LMM has worse performance in terms of goodness of fit and power. Performance in terms of fixed effect parameter recovery is equally good for both models, and in terms of type I error rate the LMM performs better than the GLMM. Finally, we provide some guidelines for applied researchers about aspects to consider when using an LMM for analyzing SCED count data.

[1]  Klaus Langohr,et al.  Parameter estimation of Poisson generalized linear mixed models based on three different statistical principles: a simulation study , 2015 .

[2]  A. W. Kemp,et al.  Univariate Discrete Distributions: Johnson/Univariate Discrete Distributions , 2005 .

[3]  Properties and comparison of estimation methods in a log-linear generalized linear mixed model , 2008 .

[4]  A. Kuk,et al.  Robust estimation in generalized linear mixed models , 2002 .

[5]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[6]  John M Ferron,et al.  Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches , 2010, Behavior research methods.

[7]  J. Gill,et al.  Generalized Linear Models: A Unified Approach , 2000 .

[8]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[9]  Bruce A. Thyer,et al.  Single-case experimental designs: Strategies for studying behavior change (2nd edn): by David H. Barlow and Michel Hersen Pergamon Press, New York, 1984, 419 pp. , 1985 .

[10]  Eden Nagler Kyse,et al.  Analyzing data from single-case designs using multilevel models: new applications and some agenda items for future research. , 2013, Psychological methods.

[11]  John M. Ferron,et al.  How Low Can You Go? An Investigation of the Influence of Sample Size and Model Complexity on Point and Interval Estimates in Two-Level Linear Models , 2014 .

[12]  Patrick Onghena,et al.  Customization of pain treatments: single-case design and analysis. , 2005, The Clinical journal of pain.

[13]  P. Onghena,et al.  Combining single-case experimental data using hierarchical linear models. , 2003 .

[14]  P. Onghena,et al.  Hierarchical linear models for the quantitative integration of effect sizes in single-case research , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[15]  William O'Donohue,et al.  How Has Applied Behavior Analysis and Behavior Therapy Changed?: An Historical Analysis of Journals. , 2007 .

[16]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[17]  P. McCullagh,et al.  Generalized Linear Models , 1984 .

[18]  Geert Molenberghs,et al.  Testing for misspecification in generalized linear mixed models. , 2010, Biostatistics.

[19]  S. Copt,et al.  Generalized Linear Models , 2001 .

[20]  Wim Van Den Noortgate,et al.  The Aggregation of Single-Case Results Using Hierarchical Linear Models. , 2007 .

[21]  G. Molenberghs,et al.  A review of generalized linear mixed models , 2002 .

[22]  C. Sachse-Lee,et al.  A Meta-Analysis of Single-Subject-Design Intervention Research for Students with LD , 2000, Journal of learning disabilities.

[23]  Alan E. Kazdin,et al.  On resolving ambiguities of the multiple-baseline design: Problems and recommendations , 1975 .

[24]  Patrick Onghena,et al.  A multilevel meta-analysis of single-subject experimental design studies , 2008 .

[25]  Mariola Moeyaert,et al.  Three-Level Analysis of Single-Case Experimental Data: Empirical Validation , 2014 .

[26]  Xiao-Hua Zhou,et al.  Statistical Computing Software Reviews Comparisons of Software Packages for Generalized Linear Multilevel Models , 1999 .

[27]  R. Winkelmann.  Econometric Analysis of Count Data , 1997 .

[28]  W. Stroup.  Generalized Linear Mixed Models: Modern Concepts, Methods and Applications , 2012 .

[29]  Satterthwaite Fe.  An approximate distribution of estimates of variance components. , 1946 .

[30]  E. Ferrer,et al.  Alternative Models for Small Samples in Psychological Research , 2016, Educational and psychological measurement.

[31]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[32]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[33]  F. Gumedze,et al.  Parameter estimation and inference in the linear mixed model , 2011 .

[34]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[35]  Andrew Gelman,et al.  Data Analysis Using Regression and Multilevel/Hierarchical Models , 2006 .

[36]  Roxana-Otilia-Sonia Hritcu,et al.  REVIEW OF JOOP J. HOX MULTILEVEL ANALYSIS – TECHNIQUES AND APPLICATIONS, Second Edition, Routledge (2010) , 2014 .

[37]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[38]  Jiming Jiang.  Linear and Generalized Linear Mixed Models and Their Applications , 2007 .

[39]  W. Shadish,et al.  The state of the science in the meta-analysis of single-case experimental designs , 2008 .

[40]  Wim Van den Noortgate,et al.  Estimation of a Nonlinear Intervention Phase Trajectory for Multiple-Baseline Design Data , 2015 .

[41]  Per B. Brockhoff,et al.  lmerTest Package: Tests in Linear Mixed Effects Models , 2017 .

[42]  D. Barlow,et al.  Single Case Experimental Designs: Strategies for Studying Behavior Change , 1976 .

[43]  A. W. Kemp,et al.  Univariate Discrete Distributions , 1993 .

[44]  W. Shadish,et al.  Characteristics of single-case designs used to assess intervention effects in 2008 , 2011, Behavior research methods.

[45]  D. Redden,et al.  Comparing denominator degrees of freedom approximations for the generalized linear mixed model in analyzing binary outcome in small sample cluster-randomized trials , 2015, BMC Medical Research Methodology.

[46]  Mithat Gönen,et al.  An assessment of estimation methods for generalized linear mixed models with binary outcomes , 2013, Statistics in medicine.

[47]  P. McCullagh,et al.  Generalized Linear Models, 2nd Edn. , 1990 .

[48]  A. W. Kemp,et al.  Univariate Discrete Distributions , 1993 .

[49]  John M Ferron,et al.  The Three-Level Synthesis of Standardized Single-Subject Experimental Data: A Monte Carlo Simulation Study , 2013, Multivariate behavioral research.