"Yours is better!": participant response bias in HCI

Although HCI researchers and practitioners frequently work with groups of people that differ significantly from themselves, little attention has been paid to the effects these differences have on the evaluation of HCI systems. Via 450 interviews in Bangalore, India, we measure participant response bias due to interviewer demand characteristics and the role of social and demographic factors in influencing that bias. We find that respondents are about 2.5x more likely to prefer a technological artifact they believe to be developed by the interviewer, even when the alternative is identical. When the interviewer is a foreign researcher requiring a translator, the bias towards the interviewer's artifact increases to 5x. In fact, the interviewer's artifact is preferred even when it is degraded to be obviously inferior to the alternative. We conclude that participant response bias should receive more attention within the CHI community, especially when designing for underprivileged populations.

[1]  V. Evers.  CROSS-CULTURAL UNDERSTANDING OF METAPHORS IN INTERFACE DESIGN , 1998 .

[2]  S. Asch.  Effects of Group Pressure Upon the Modification and Distortion of Judgments , 1951 .

[3]  Elizabeth F Loftus,et al.  The Red Herring technique: a methodological response to the problem of demand characteristics , 2008, Psychological research.

[4]  D. Paulhus.  Measurement and control of response bias. , 1991 .

[5]  John P. Robinson,et al.  Measures Of Personality And Social Psychological Attitudes , 1991 .

[6]  Jonathan Ledlie.  “ Huzzah for my thing : ” Evaluating a Pilot of a Mobile Service in Kenya , 2010 .

[7]  Fff Simona Nnn Bignami Van Assche,et al.  An Assessment of the KDICP and MDICP Data Quality , 2003 .

[8]  Barry A. T. Brown,et al.  Into the wild: challenges and opportunities for field trial methods , 2011, CHI.

[9]  Matthew Kam,et al.  Human computer interaction for international development: past present and future , 2011 .

[10]  Alexander A. Weinreb,et al.  The Limitations of Stranger-Interviewers in Rural Kenya , 2006 .

[11]  Lilly Irani.  HCI on the move: methods, culture, values , 2010, CHI EA '10.

[12]  Alan G. Sawyer.  Detecting Demand Characteristics in Laboratory Experiments in Consumer Research: the Case of Repetition-Affect Research , 1975 .

[13]  Wendy E. Mackay,et al.  Triangulation within and across HCI disciplines , 1999 .

[14]  Thomas N. Smyth,et al.  Notes from the Field Stories from the Field : Reflections on HCI 4 D Experiences , 2009 .

[15]  Mary Czerwinski,et al.  Subjective Duration Assessment: An Implicit Probe for Software Usability , 2001 .

[16]  Manuel A. Pérez-Quiñones,et al.  Culture and usability evaluation: the effects of culture in structured interviews , 2006 .

[17]  Austin Lee Nichols,et al.  The Good-Subject Effect: Investigating Participant Demand Characteristics , 2008, The Journal of general psychology.

[18]  Richard E. Ladner,et al.  Usable gestures for blind people: understanding preference and performance , 2011, CHI.

[19]  J. Williams,et al.  INTERVIEWER ROLE PERFORMANCE: A FURTHER NOTE ON BIAS IN THE INFORMATION INTERVIEW , 1968 .

[20]  M. Bernhart,et al.  Patient satisfaction in developing countries. , 1999, Social science & medicine.

[21]  Vanessa Evers,et al.  The Role of Culture in Interface Acceptance , 1997, INTERACT.

[22]  T. Johnson,et al.  Interviewer effects on self-reported substance use among homeless persons. , 1994, Addictive behaviors.

[23]  C H Caldwell,et al.  Interviewer effects in public health surveys. , 2010, Health education research.

[24]  Erich Kirchler,et al.  The influence of member status differences and task type on group consensus and member position change , 1986 .

[25]  Daniel John Zizzo,et al.  Experimenter demand effects in economic experiments , 2008 .

[26]  Fay Sudweeks,et al.  Cultural attitudes towards technology and communication , 2000 .

[27]  Jerry Suls,et al.  More on the social psychology of the experiment: When compliance turns to self-defense. , 1973 .

[28]  F. Strodtbeck,et al.  Becoming first among equals: Moral considerations in jury foreman selection. , 1985 .

[29]  Morten Hertzum,et al.  The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments , 1998 .

[30]  M. Orne.  On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. , 1962 .

[31]  S. Milgram.  BEHAVIORAL STUDY OF OBEDIENCE. , 1963, Journal of abnormal psychology.

[32]  H. Guetzkow,et al.  Groups, Leadership, and Men , 1952 .