Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field

The user study is a fundamental method used in HCI. In designing user studies, we often use compensation strategies to incentivize recruitment. However, compensation can also lead to ethical issues, such as coercion. The CHI community has yet to establish best practices for participant compensation. Through a systematic review of manuscripts at CHI and other associated publication venues, we found high levels of variation in the compensation strategies used within the community and how we report on this aspect of the study methods. A qualitative analysis of justifications offered for compensation sheds light into how some researchers are currently contextualizing this practice. This paper provides a description of current compensation strategies and information that can inform the design of compensation strategies in future studies. The findings may be helpful to generate productive discourse in the HCI community towards the development of best practices for participant compensation in user studies.

[1]  Babak Naderi,et al.  Who are the Crowdworkers , 2018 .

[2]  D. Freed.  Analyzing Clinical Computer Security Interventions with Survivors of Intimate Partner Violence , 2019 .

[3]  J. A. van Biljon,et al.  A critical review on the reporting of surveys in transdisciplinary research: A case study in Information Systems , 2011 .

[4]  Patrick Olivier,et al.  Configuring participation: on how we involve people in design , 2013, CHI.

[5]  Mary Czerwinski,et al.  Pocket Skills: A Conversational Mobile Web App To Support Dialectical Behavioral Therapy , 2018, CHI.

[6]  Per Ola Kristensson,et al.  Crowdworker Economics in the Gig Economy , 2019, CHI.

[7]  Matthew Podolsky,et al.  The open incentive kit (OINK): standardizing the generation, comparison, and deployment of incentive systems , 2019, ICTD.

[8]  Niklas Elmqvist,et al.  Ranked-List Visualization: A Graphical Perception Study , 2019, CHI.

[9]  Darren Guinness,et al.  Caption Crawler: Enabling Reusable Alternative Text Descriptions using Reverse Image Search , 2018, CHI.

[10]  Tawanna Dillahunt,et al.  Online Grocery Delivery Services: An Opportunity to Address Food Disparities in Transportation-scarce Areas , 2019, CHI.

[11]  K. Beck.  Academic researcher decision making processes for research participant compensation , 2019 .

[12]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[13]  Wai-Tat Fu,et al.  A Player-Centric Approach to Designing Spatial Skill Training Games , 2019, CHI.

[14]  Mary L. Gray,et al.  Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass , 2019 .

[15]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[16]  Anne Marie Piper,et al.  "It doesn't win you friends" , 2019, Proc. ACM Hum. Comput. Interact..

[17]  Brad A. Myers,et al.  A brief history of human-computer interaction technology , 1998, INTR.

[18]  A. Dellinger.  Subjected to Science: Human Experimentation in America Before the Second World War , 1996 .

[19]  M. Six Silberman,et al.  Turkopticon: interrupting worker invisibility in amazon mechanical turk , 2013, CHI.

[20]  Prateek Jain,et al.  GesturePod: Enabling On-device Gesture-based Interaction for White Cane Users , 2019, UIST.

[21]  Bengt Sandblad,et al.  Human-Computer Interaction (HCI) , 2013 .

[22]  Elizabeth D. Mynatt,et al.  Exploring Indicators of Digital Self-Harm with Eating Disorder Patients , 2019, Proc. ACM Hum. Comput. Interact..

[23]  Meredith Ringel Morris,et al.  Crowdsourcing Similarity Judgments for Agreement Analysis in End-User Elicitation Studies , 2018, UIST.

[24]  R. Betensky,et al.  Research participant compensation: A matter of statistical inference as well as ethics. , 2015, Contemporary clinical trials.

[25]  Haiyi Zhu,et al.  Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders , 2019, CHI.

[26]  Nabil Alshurafa,et al.  Is More Always Better?: Discovering Incentivized mHealth Intervention Engagement Related to Health Behavior Trends , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[27]  Aaron Steinfeld,et al.  Speak Up: A Multi-Year Deployment of Games to Motivate Speech Therapy in India , 2018, CHI.

[28]  Jacki O'Neill,et al.  Being a turker , 2014, CSCW.

[29]  Nicola Dell,et al.  “A Stalker's Paradise”: How Intimate Partner Abusers Exploit Technology , 2018, CHI.

[30]  M. Strohmaier,et al.  Measuring Motivations of Crowdworkers: The Multidimensional Crowdworker Motivation Scale , 2017 .

[31]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[32]  Anna L. Cox,et al.  Monotasking or Multitasking: Designing for Crowdworkers' Preferences , 2019, CHI.

[33]  Ranjitha Kumar,et al.  How do People Sort by Ratings? , 2019, CHI.

[34]  Rachel Greenstadt,et al.  Privacy, Anonymity, and Perceived Risk in Open Collaboration: A Study of Service Providers , 2019, CHI.

[35]  Priya Kumar,et al.  'I Knew It Was Too Good to Be True" , 2018, Proc. ACM Hum. Comput. Interact..

[36]  Jeffrey Heer,et al.  Value-Suppressing Uncertainty Palettes , 2018, CHI.

[37]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[38]  Claus Bossen,et al.  Participatory design in an era of participation , 2017 .

[39]  Richard J. Anderson,et al.  ReCall: Crowdsourcing on Basic Phones to Financially Sustain Voice Forums , 2019, CHI.

[40]  Gary M Olson,et al.  Human-computer interaction: psychological aspects of the human use of computing. , 2003, Annual review of psychology.

[41]  Nitesh V. Chawla,et al.  Differentiating Higher and Lower Job Performers in the Workplace Using Mobile Sensing , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[42]  Nicola Dell,et al.  "Is my phone hacked?" Analyzing Clinical Computer Security Interventions with Survivors of Intimate Partner Violence , 2019, Proc. ACM Hum. Comput. Interact..

[43]  Tim Kraska,et al.  VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository , 2019, CHI.

[44]  Ioannis Arapakis,et al.  Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction , 2011, Inf. Process. Manag..

[45]  Regan L. Mandryk,et al.  Let Me Be Implicit: Using Motive Disposition Theory to Predict and Explain Behaviour in Digital Games , 2018, CHI.

[46]  Isaac Wang,et al.  Exploring Virtual Agents for Augmented Reality , 2019, CHI.

[47]  Katie Shilton,et al.  Beyond the Belmont Principles: Ethical Challenges, Practices, and Beliefs in the Online Data Research Community , 2016, CSCW.

[48]  Wai-Tat Fu,et al.  VidLyz: An Interactive Approach to Assist Novice Entrepreneurs in Making Persuasive Campaign Videos , 2019, Proc. ACM Hum. Comput. Interact..

[49]  C. Grady,et al.  An analysis of U.S. practices of paying research participants. , 2005, Contemporary clinical trials.

[50]  Min Zhang,et al.  Designing for the Infrastructure of the Supply Chain of Malay Handwoven Songket in Terengganu , 2019, CHI.

[51]  Wendy E. Mackay,et al.  Knotation: Exploring and Documenting Choreographic Processes , 2018, CHI.

[52]  Sean A. Munson,et al.  Beacon , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[53]  Christopher A. Le Dantec,et al.  "The cavalry ain't coming in to save us" , 2019, Proc. ACM Hum. Comput. Interact..

[54]  Chris Callison-Burch,et al.  A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk , 2017, CHI.

[55]  Elena Paslaru Bontas Simperl,et al.  Social Incentives in Paid Collaborative Crowdsourcing , 2017, ACM Trans. Intell. Syst. Technol..

[56]  Stephen A. Brewster,et al.  Evaluation of Haptic Patterns on a Steering Wheel , 2016, AutomotiveUI.

[57]  Casey Fiesler,et al.  “Participant” Perceptions of Twitter Research Ethics , 2018 .

[58]  Niels Henze,et al.  Using Presence Questionnaires in Virtual Reality , 2019, CHI.

[59]  Walter S. Lasecki,et al.  Arboretum and Arbility: Improving Web Accessibility Through a Shared Browsing Architecture , 2018, UIST.

[60]  Marc Hassenzahl,et al.  Changing Perspective: A Co-Design Approach to Explore Future Possibilities of Divergent Hearing , 2019, CHI.

[61]  P. Appelbaum,et al.  The Reporting of Monetary Compensation in Research Articles , 2007, Journal of empirical research on human research ethics : JERHRE.

[62]  Meredith Ringel Morris,et al.  Understanding the Needs of Searchers with Dyslexia , 2018, CHI.

[63]  Jacob Thebault-Spieker,et al.  GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations , 2019, Proc. ACM Hum. Comput. Interact..

[64]  Nicole S. Goedhart,et al.  Exploring the boundaries of ‘good’ Participatory Action Research in times of increasing popularity: dealing with constraints in local policy for digital inclusion , 2020 .

[65]  J. Merz,et al.  How Much Are Subjects Paid to Participate in Research? , 2001, The American journal of bioethics : AJOB.

[66]  Carman Neustaedter,et al.  Shopping Over Distance through a Telepresence Robot , 2018, Proc. ACM Hum. Comput. Interact..

[67]  C. Grady,et al.  Payment of clinical research subjects. , 2005, The Journal of clinical investigation.

[68]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[69]  Coye Cheshire,et al.  Incentives to participate in online research: an experimental examination of "surprise" incentives , 2014, CHI.

[70]  Nicole Martino,et al.  Commonly performed procedures in clinical research: a benchmark for payment. , 2012, Contemporary clinical trials.

[71]  Alexis Hiniker,et al.  Modeling the Engagement-Disengagement Cycle of Compulsive Phone Use , 2019, CHI.

[72]  Kar-Hai Chu,et al.  Re-evaluating standards of human subjects protection for sensitive health data in social media networks , 2019, Soc. Networks.

[73]  Weiyu Zhang,et al.  Effects of Moderation and Opinion Heterogeneity on Attitude towards the Online Deliberation Experience , 2019, CHI.

[74]  Kelly Caine,et al.  Local Standards for Sample Size at CHI , 2016, CHI.

[75]  Donghee Yvette Wohn,et al.  Individual and Collaborative Behaviors of Rideshare Drivers in Protecting their Safety , 2019, Proc. ACM Hum. Comput. Interact..

[76]  Colin Scott,et al.  Exploring Crowdsourced Work in Low-Resource Settings , 2019, CHI.

[77]  G. Henderson.  The ethics of HIV "cure" research: what can we learn from consent forms? , 2015, AIDS research and human retroviruses.

[78]  C. Grady,et al.  What's the price of a research subject? Approaches to payment for research participation. , 1999, The New England journal of medicine.

[79]  Andrew Raij,et al.  Exploring micro-incentive strategies for participant compensation in high-burden studies , 2011, UbiComp '11.

[80]  Jacob O. Wobbrock,et al.  Cluster Touch: Improving Touch Accuracy on Smartphones for People with Motor and Situational Impairments , 2019, CHI.

[81]  Frank Bentley,et al.  Understanding Online News Behaviors , 2019, CHI.

[82]  E. Largent,et al.  Paying Research Participants: Regulatory Uncertainty, Conceptual Confusion, and a Path Forward. , 2018, Yale journal of health policy, law, and ethics.

[83]  BEN GREEN,et al.  The Principles and Limits of Algorithm-in-the-Loop Decision Making , 2019, Proc. ACM Hum. Comput. Interact..

[84]  C. Gennings,et al.  Why Do We Pay? A National Survey of Investigators and IRB Chairpersons , 2010, Journal of empirical research on human research ethics : JERHRE.

[85]  Tom Feltwell,et al.  Rethinking Engagement with Online News through Social and Visual Co-Annotation , 2018, CHI.