Using Nonnaive Participants Can Reduce Effect Sizes

Although researchers often assume their participants are naive to experimental materials, this is not always the case. We investigated how prior exposure to a task affects subsequent experimental results. Participants in this study completed the same set of 12 experimental tasks at two points in time, first as a part of the Many Labs replication project and again a few days, a week, or a month later. Effect sizes were markedly lower in the second wave than in the first. The reduction was most pronounced when participants were assigned to a different condition in the second wave. We discuss the methodological implications of these findings.

[1]  David G. Rand,et al.  Social heuristics shape intuitive cooperation , 2014, Nature Communications.

[2]  Thomas D. Cook,et al.  Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. , 1972 .

[3]  A. Tversky,et al.  The framing of decisions and the psychology of choice. , 1981, Science.

[4]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[5]  Duncan J. Watts,et al.  Long-run learning in games of cooperation , 2014, EC.

[6]  Richard J. Crisp,et al.  Elaboration enhances the imagined contact effect , 2010 .

[7]  Irving Lorge,et al.  Prestige, suggestion, and attitudes. , 1936 .

[8]  Craig R. Fox,et al.  Ambiguity Aversion and Comparative Ignorance , 1995 .

[9]  Daniel M. Oppenheimer,et al.  Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power , 2009 .

[10]  Gary Charness,et al.  Journal of Economic Behavior & Organization , 2022 .

[11]  Marcia K. Johnson,et al.  Source monitoring. , 1993, Psychological bulletin.

[12]  P. Schoemaker,et al.  Prospect theory's reflection hypothesis: A critical examination , 1980 .

[13]  S. Sherman.  On the self-erasing nature of errors of prediction. , 1980 .

[14]  Reginald B. Adams,et al.  Investigating Variation in Replicability: A “Many Labs” Replication Project , 2014 .

[15]  Michael H. Birnbaum,et al.  How to show that 9 > 221 : Collect judgments in a between-subjects design , 1999 .

[16]  Norbert Schwarz,et al.  Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments , 1985 .

[17]  Donald D. Rugg,et al.  EXPERIMENTS IN WORDING QUESTIONS: II , 1941 .

[18]  Steven J. Sherman,et al.  The role of prior knowledge in explaining hypothetical events , 1985 .

[19]  D. Sunshine Hillygus,et al.  Professional respondents in nonprobability online panels , 2014 .

[20]  D. Kahneman,et al.  Measures of Anchoring in Estimation Tasks , 1995 .

[21]  Patrick Sturgis,et al.  Attitudes Over Time: The Psychology of Panel Conditioning , 2009 .

[22]  A. Greenwald,et al.  Measuring individual differences in implicit cognition: the implicit association test. , 1998, Journal of personality and social psychology.

[23]  Brian A. Nosek,et al.  Health of the Implicit Association Test at age 3. , 2001, Zeitschrift fur experimentelle Psychologie : Organ der Deutschen Gesellschaft fur Psychologie.

[24]  J. Goyder,et al.  Surveys on Surveys: Limitations and Potentialities , 1986 .

[25]  Jesse Chandler,et al.  Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers , 2013, Behavior Research Methods.

[26]  C. F. Bond,et al.  One Hundred Years of Social Psychology Quantitatively Described , 2003 .

[27]  Daniel M. Oppenheimer,et al.  The retrospective gambler’s fallacy: Unlikely events, constructing the past, and multiple universes , 2009, Judgment and Decision Making.

[28]  A. Greenwald.  Within-subjects designs: To use or not to use? , 1976 .

[29]  E. Erdfelder,et al.  Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses , 2009, Behavior research methods.