Examining the Rule of Thumb of Not Using Multilevel Modeling: The “Design Effect Smaller Than Two” Rule

Educational researchers commonly use the rule of thumb of “design effect smaller than 2” as the justification of not accounting for the multilevel or clustered structure in their data. The rule, however, has not yet been systematically studied in previous research. In the present study, we generated data from three different models (which differ in the location of the clustering effect). With a 3 (design effect) × 5 (cluster size) × 4 (number of clusters) Monte Carlo simulation study we found that the rule should not be applied when researchers: (a) are interested in the effects of higher-level predictors, or (b) have a cluster size less than 10. Implications of the findings and limitations of the study are discussed.

[1]  S. Sheridan,et al.  Family involvement for children with disruptive behaviors: the role of parenting stress and motivational beliefs. , 2010, Journal of school psychology.

[2]  J. Hox.  Multilevel analysis: Techniques and applications, 2nd ed. , 2010 .

[3]  Qing Zhou,et al.  Teacher-child relationship quality and academic achievement of Chinese American children in immigrant families. , 2012, Journal of school psychology.

[4]  Cora J. M. Maas,et al.  The influence of violations of assumptions on multilevel parameter estimates and their standard errors , 2004, Comput. Stat. Data Anal..

[5]  T. Pettigrew,et al.  Prejudice And Minority Proportion: Contact Instead Of Threat Effects , 2006 .

[6]  Renate von Grünigen,et al.  Links between local language competence and peer relations among Swiss and immigrant children: the mediating role of social behavior. , 2012, Journal of school psychology.

[7]  Roel Bosker,et al.  Standard Errors and Sample Sizes for Two-Level Research , 1993 .

[8]  J. Hox,et al.  Sufficient Sample Sizes for Multilevel Modeling , 2005 .

[9]  Toni Kempler Rogat,et al.  Affect and engagement during small group instruction , 2011 .

[10]  M. Hofer,et al.  Value orientations as determinants and outcomes of conflicts between on-task and off-task actions in the classroom , 2010 .

[11]  Harvey Goldstein,et al.  League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance , 1996 .

[12]  M. Vermande,et al.  Peer and self-reports of victimization and bullying: their differential association with internalizing problems and social adjustment. , 2012, Journal of school psychology.

[13]  B. Muthén,et al.  Multilevel Covariance Structure Analysis , 1994 .

[14]  Anne Buu,et al.  Trajectories of childhood aggression and inattention/hyperactivity: differential effects on substance abuse in adolescence. , 2008, Journal of the American Academy of Child and Adolescent Psychiatry.

[15]  A. Boomsma,et al.  Robustness Studies in Covariance Structure Modeling , 1998 .

[16]  Robert S. Stawski,et al.  Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling (2nd Edition) , 2013 .

[17]  Anees Janee Ali,et al.  Determinants of intercultural adjustment among expatriate spouses , 2003 .

[18]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[19]  E. Youngstrom,et al.  Informant discrepancies in clinical reports of youths and interviewers' impressions of the reliability of informants. , 2011, Journal of child and adolescent psychopharmacology.

[20]  F. Goossens,et al.  Parental strategies and trajectories of peer victimization in 4 to 5 year olds. , 2011, Journal of school psychology.

[21]  A. Satorra,et al.  Complex Sample Data in Structural Equation Modeling , 1995 .

[22]  Jon Rasbash,et al.  Cross-Classified and Multiple Membership Models† , 2014 .

[23]  K F Rust,et al.  Variance estimation for complex surveys using replication techniques , 1996, Statistical methods in medical research.

[24]  Samuel B. Green,et al.  The Impact of Misspecifying the Within-Subject Covariance Structure in Multiwave Longitudinal Multilevel Models: A Monte Carlo Study , 2007 .

[25]  Understanding Latino Children's Heterogeneous Academic Growth Trajectories: Latent Growth Mixture Modeling Approach , 2012 .

[26]  Kenneth Adams,et al.  Behavioral control and resiliency in the onset of alcohol and illicit drug use: a prospective study from preschool to adolescence. , 2006, Child development.

[27]  E. Leeuw,et al.  The See-Saw Effect: a multilevel problem? , 1988 .

[28]  Eric M. Anderman,et al.  The classroom environment and students' reports of avoidance strategies in mathematics: A multimethod study. , 2002 .

[29]  William J. Browne,et al.  Implementation and performance issues in the Bayesian and likelihood fitting of multilevel models , 2000, Comput. Stat..

[30]  J. Hox,et al.  Sample sizes for multilevel modeling , 2002 .

[31]  Yulin Fang,et al.  Socialization in Open Source Software Projects: A Growth Mixture Modeling Approach , 2011 .

[32]  Ehri Ryu,et al.  Family Processes Mediating the Relationship of Neighborhood Disadvantage to Early Adolescent Internalizing Problems , 2006 .

[33]  James L Peugh,et al.  A practical guide to multilevel modeling. , 2010, Journal of school psychology.

[34]  Kosuke Imai,et al.  Survey Sampling , 1998, Nov/Dec 2017.

[35]  Lisa M LaVange,et al.  Innovative strategies using SUDAAN for analysis of health surveys with complex samples , 1996, Statistical methods in medical research.

[36]  T. Hart-Johnson,et al.  The association among neighborhood socioeconomic status, race and chronic pain in black and white older adults. , 2007, Journal of the National Medical Association.

[37]  P. Clarke,et al.  When can group level clustering be ignored? Multilevel models versus single-level models with sparse data , 2008, Journal of Epidemiology & Community Health.

[38]  Craig K. Enders,et al.  Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. , 2007, Psychological methods.

[39]  R. Zucker,et al.  Self-concept disturbances: cognitive vulnerability for early drinking and early drunkenness in adolescents at high risk for alcohol problems. , 2008, Addictive behaviors.

[40]  Scott L. Thomas,et al.  Analysis of Large-Scale Secondary Data in Higher Education Research: Potential Perils Associated with Complex Sampling Designs , 2001 .