Setting Doesn’t Matter Much

Abstract. This study deals with the effects of the diagnostic setting on the performance in intelligence tests. We conducted a meta-analysis in which k = 30 samples with a total sample size of N = 2,448 were integrated. We compared results for the same intelligence tests administered either in a group or in an individual setting. The main analysis indicated a small mean population effect [ M( g) = 0.085] that was not significant [−0.036 ≤  M( g) ≤ 0.206]. Nevertheless, moderator analyses indicated a stronger [ M( g) = 0.193] and significant [0.087 ≤  M( g) ≤ 0.298] effect in favor of individual settings for studies employing a between-person design. Setting effects in within-person designs were most likely superimposed by retest effects. As the setting effect was very small, the current testing practice in which results obtained in group and individual settings are treated as interchangeable is not overly problematic. However, our results encourage test developers to examine setting effects before stating that results obtained in different settings are equivalent. Between-person designs using participants of comparable ability are most suitable in this context as retest effects can be ruled out.

[1]  D. Altman,et al.  Measuring inconsistency in meta-analyses , 2003, BMJ : British Medical Journal.

[2]  Note concerning group influence upon Otis S-A test scores. , 1936 .

[3]  R. W. Husband.  Analysis of methods in human maze learning , 1929 .

[4]  W. Reitman.  Motivational induction and the behavior correlates of the achievement and affiliation motives. , 1960, Journal of abnormal and social psychology.

[5]  N. Triplett,et al.  The Dynamogenic Factors in Pacemaking and Competition , 1898 .

[6]  Winston Bennett,et al.  Conducting Meta-Analysis Using SAS , 2001 .

[7]  H. Moore.  Laboratory Tests of Anger, Fear and Sex Interest , 1917 .

[8]  Joseph Pessin,et al.  The Comparative Effects of Social and Mechanical Stimulation on Memorizing , 1933 .

[9]  Wolfgang Viechtbauer,et al.  Conducting Meta-Analyses in R with the metafor Package , 2010 .

[10]  Masayuki Henmi,et al.  Confidence intervals for random effects meta‐analysis and robustness to publication bias , 2010, Statistics in medicine.

[11]  Philip Schatz,et al.  Group Versus Individual Administration Affects Baseline Neurocognitive Test Performance , 2011, The American journal of sports medicine.

[12]  The Influence of the Group on Psychological Test Scores , 1926 .

[13]  Group Effects on Reasoning Functions , 1952 .

[14]  Joan J. Michael,et al.  The Predictive Validity of the Developmental Test of Visual-Motor Integration Under Group and Individual Modes of Administration Relative to Academic Performance Measures of Second-Grade Pupils Without Identifiable Major Learning Disabilities , 1979 .

[15]  Franziska Lemke,et al.  Comparison of ability tests administered online and in the laboratory , 2009, Behavior research methods.

[16]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[17]  D. J. Goldstein.  Group and Individual Administration of the Wickens Technique , 1978 .

[18]  C. Anderson.  An Experimental Study of "Social Facilitation" as Affected by "Intelligence" , 1929, American Journal of Sociology.

[19]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[20]  A group-administered lag task as a measure of working memory , 2007, Behavior research methods.

[21]  J. F. Dashiell.  An experimental analysis of some group effects. , 1930 .

[22]  N. Milgram,et al.  Group versus Individual Administration in the Measurement of Creative Thinking in Gifted and Nongifted Children , 1976 .

[23]  C. F. Bond,et al.  Social facilitation: a meta-analysis of 241 studies. , 1983, Psychological bulletin.

[24]  Frédéric Dandurand,et al.  Comparing online and lab methods in a problem-solving experiment , 2008, Behavior research methods.

[25]  J. Sterne,et al.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. , 2000, Journal of clinical epidemiology.

[26]  R. Norris,et al.  Comparability of Peabody Picture Vocabulary Test scores under group and individual administration. , 1960 .

[27]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[28]  G. Larson,et al.  Two Modes of Administration of the Bender Visual-Motor Gestalt Test to Kindergarten Children , 1977, Perceptual and motor skills.

[29]  Group administration of the Bender Gestalt , 1971 .

[30]  G. Brannigan,et al.  Comparison of Individual versus Group Administration of the Modified Version of the Bender-Gestalt Test , 1995, Perceptual and motor skills.

[31]  F. Allport.  The influence of the group upon association and thought. , 1920 .

[32]  G. Rücker,et al.  Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. , 2011, Biostatistics.

[33]  G A Fargo,et al.  Comparability of group television and individual administration of the Peabody Picture Vocabulary Test: implications for screening. , 1967, Journal of educational psychology.

[34]  A. Luchins,et al.  Geometric problem solving related to differences in sex and mathematical interests. , 1979, The Journal of genetic psychology.

[35]  John E. Hunter,et al.  Methods of Meta-Analysis , 1989 .

[36]  Bernd Strauss,et al.  Social facilitation in motor tasks: a review of research and theory , 2002 .

[37]  Factors influencing performance on group and individual tests of intelligence; social facilitation. , 1946, Journal of educational psychology.

[38]  Paul R. Farnsworth.  Concerning So-Called Group Effects , 1928 .

[39]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[40]  R. Zajonc.  SOCIAL FACILITATION. , 1965, Science.

[41]  W. M. Cox,et al.  Individual versus Group Administration of the Memory-For-Designs Test to Alcoholics , 1999 .

[42]  J. Ireland.  A comparison of individual and group administration of the Cattell Culture Fair Intelligence Scale and the Weschler Intelligence Scale for Children in a deliquent population , 1974 .

[43]  F. Preckel,et al.  Online- versus paper-pencil-version of a high potential intelligence test , 2003 .

[44]  Performance of Norwegian Children on the Bender Gestalt and Draw‐a‐Person Tests , 1970 .