Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys

Instructions and virtual presence were used to prevent careless responding in online surveys.MANOVAs showed warning instructions reduced some forms of careless responding.Virtual presence enhances the utility of survey instructions in preventing careless responding.Feedback instructions hold promise as a non-aversive way to prevent careless responding. Internet-based survey data inform knowledge creation in research and justify work activities in organizations. While there are advantages to online surveys, this mode of administration comes with its own set of challenges. Survey respondents may engage in careless responding (i.e. insufficient effort responding or satisficing) by intentionally or unintentionally responding in a manner that does not accurately reflect their true sentiments. Careless responding can create psychometric problems even after correctly removing careless respondents (i.e. mischievous responders). This study aimed to improve survey methodology by preventing careless responding. Using a 3i?3 between-subjects experimental design, we manipulated both virtual presence (none, animated shape, and virtual human) and type of instructions (anonymous, warning, and feedback). Indicators of careless responding were the dependent variables. Results showed that beyond characteristics of survey items, survey design elements can prevent careless responding. The effects of interventions differed by type of careless responding. Instructions, and the interaction of instructions and virtual presence significantly reduced careless responding, but not virtual presence alone. Virtual human presence increased the salience of instructions. Although currently effective, instructions warning of punitive consequences may create difficulty in recruiting participants. Future research should continue investigating non-aversive ways to prevent careless responding on Internet-based surveys.

[1]  Richard Catrambone,et al.  Social Facilitation Effects of Virtual Humans , 2007, Hum. Factors.

[2]  J. Haxby,et al.  Human neural systems for face recognition and social communication , 2002, Biological Psychiatry.

[3]  Alice Hall,et al.  What's going on? Age, distraction, and multitasking during online survey taking , 2014, Comput. Hum. Behav..

[4]  J. Krosnick.  Response strategies for coping with the cognitive demands of attitude measures in surveys , 1991 .

[5]  A. Meade,et al.  Identifying careless responses in survey data. , 2012, Psychological methods.

[6]  Larry D. Rosen,et al.  Multitasking across generations: Multitasking choices and difficulty ratings in three generations of Americans , 2009, Comput. Hum. Behav..

[7]  B. Tabachnick,et al.  Using multivariate statistics, 5th ed. , 2007 .

[8]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[9]  N. Schwarz.  Self-reports: How the questions shape the answers. , 1999 .

[10]  H. Baumgartner,et al.  1 Running head : Grouping Survey Items and Discriminant Validity Discriminant Validity Where There Should Be None : Positioning Same-Scale Items in Separated Blocks of a Questionnaire , 2015 .

[11]  K A Kiehl,et al.  Assessing psychopathic attributes in a noninstitutionalized population. , 1995, Journal of personality and social psychology.

[12]  R. Catrambone,et al.  Social Responses to Virtual Humans: The Effect of Human-Like Characteristics , 2021, Applied Sciences.

[13]  Arthur H. Miller,et al.  American national election studies data sourcebook, 1952-1978 , 1981 .

[14]  C. D. De Dreu,et al.  Self-interest and other-orientation in organizational behavior: implications for job performance, prosocial behavior, and personal initiative. , 2009, The Journal of applied psychology.

[15]  Carol M. Woods.  Careless Responding to Reverse-Worded Items: Implications for Confirmatory Factor Analysis , 2006 .

[16]  William Sims Bainbridge,et al.  Leadership in science and technology : a reference handbook , 2012 .

[17]  Neal Schmitt,et al.  Factors Defined by Negatively Keyed Items: The Result of Careless Respondents? , 1985 .

[18]  John P. Robinson,et al.  Measures of social psychological attitudes , 1976 .

[19]  R L Greene,et al.  Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials. , 1989, Journal of clinical psychology.

[20]  Tara S. Behrend,et al.  Similarity effects in online training: Effects with computerized trainer agents , 2011, Comput. Hum. Behav..

[21]  P. Hardré,et al.  Examining Contexts-of-Use for Web-Based and Paper-Based Questionnaires , 2012 .

[22]  William K. Balzer,et al.  Development of a Compact Measure of Job Satisfaction: The Abridged Job Descriptive Index , 2002 .

[23]  Tara S. Behrend,et al.  Using Animated Agents in Learner‐Controlled Training: The Effects of Design Control , 2012 .

[24]  John A. Johnson.  Ascertaining the validity of individual protocols from Web-based personality inventories. , 2005 .

[25]  D. Paulhus.  Two-component models of socially desirable responding. , 1984 .

[26]  Gregory B Northcraft,et al.  Feedback and the rationing of time and effort among competing tasks. , 2011, The Journal of applied psychology.

[27]  J. P. Robinson-Cimpian,et al.  Inaccurate Estimation of Disparities Due to Mischievous Responders , 2014 .

[28]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[29]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[30]  R. Raskin,et al.  A principal-components analysis of the Narcissistic Personality Inventory and further evidence of its construct validity. , 1988, Journal of personality and social psychology.

[31]  S. Toaddy.  Validation of a Measure of External Organizational Justice , 2012 .

[32]  Terri L. Griffith.  Monitoring and Performance: A Comparison of Computer and Supervisor Monitoring , 1993 .

[33]  Stephen M. Colarelli,et al.  Comparative effects of personal and situational influences on job outcomes of new professionals. , 1987 .

[34]  R. Baer,et al.  MMPI-2 Random Responding Indices: Validation Using a Self-Report Methodology. , 1992 .

[35]  John R. Aiello,et al.  The effect of challenge and threat appraisals under evaluative presence. , 2010 .

[36]  John R. Aiello,et al.  Social facilitation: A test of competing theories , 2006 .

[37]  Hunter Gehlbach,et al.  Anchoring and Adjusting in Questionnaire Responses , 2012 .

[38]  Shauna P. Acquavita,et al.  Personal and Organizational Diversity Factors' Impact on Social Workers' Job Satisfaction: Results from a National Internet-Based Survey , 2009 .

[39]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[40]  M. Black.  Avatars , 2008, BMJ : British Medical Journal.

[41]  Ning Wang,et al.  Can Virtual Humans Be More Engaging Than Real Ones? , 2007, HCI.

[42]  J. Bermúdez,et al.  Personality Psychology in Europe , 1997 .

[43]  Byron Reeves,et al.  The effects of animated characters on anxiety, task performance, and evaluations of user interfaces , 2000, CHI.

[44]  John P. Robinson,et al.  Measures of political attitudes , 1970 .

[45]  Larry F. Hodges,et al.  Effects of Virtual Human Presence on Task Performance , 2004 .

[46]  Elizabeth M. Poposki,et al.  Detecting and Deterring Insufficient Effort Responding to Surveys , 2012 .

[47]  J. R. Aiello,et al.  Social facilitation from Triplett to electronic performance monitoring. , 2001 .

[48]  J. R. Aiello,et al.  Computer Monitoring of Work Performance: Extending the Social Facilitation Framework to Electronic Presence1 , 1993 .

[49]  Panagiotis G. Ipeirotis,et al.  Running Experiments on Amazon Mechanical Turk , 2010, Judgment and Decision Making.

[50]  J. L. Price.  Handbook of Organizational Measurement , 1975 .

[51]  J. Krosnick,et al.  Comparing questions with agree/ disagree response options to questions with item-specific response options , 2010 .

[52]  P. Costa,et al.  Validation of the five-factor model of personality across instruments and observers. , 1987, Journal of personality and social psychology.

[53]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[54]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[55]  J. Patrick,et al.  Being in one's chosen job determines pre-training attitudes and training outcomes , 2012 .

[56]  Hunter Gehlbach,et al.  Using the Theory of Satisficing to Evaluate the Quality of Survey Data , 2011, Research in Higher Education.

[57]  Anne Bruce,et al.  Motivating employees , 1998 .

[58]  J. Marx,et al.  Nonprofit Governance: Improving Performance in Troubled Economic Times , 2012 .

[59]  Peter Whitman,et al.  A misuse of power , 2004 .

[60]  D. Marlowe,et al.  A new scale of social desirability independent of psychopathology. , 1960, Journal of consulting psychology.

[61]  Nicole English,et al.  Prospects and Limitations of Psychological Testing on the Internet , 2002 .

[62]  David E. Wiley,et al.  Socially Desirable Responding: The Evolution of a Construct , 2001 .

[63]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[64]  L. Porter,et al.  The Measurement of Organizational Commitment. , 1979 .

[65]  N. Anderson,et al.  Social Facilitation and Inhibition During Maximum Versus Typical Performance Situations , 2007 .