Effective Strategies for Crowd-Powered Cognitive Reappraisal Systems: A Field Deployment of the Flip*Doubt Web Application for Mental Health

Online technologies offer great promise to expand models of delivery for therapeutic interventions to help users cope with increasingly common mental illnesses like anxiety and depression. For example, "cognitive reappraisal" is a skill that involves changing one's perspective on negative thoughts in order to improve one's emotional state. In this work, we present Flip*Doubt, a novel crowd-powered web application that provides users with cognitive reappraisals ("reframes") of negative thoughts. A one-month field deployment of Flip*Doubt with 13 graduate students yielded a data set of negative thoughts paired with positive reframes, as well as rich interview data about how participants interacted with the system. Through this deployment, our work contributes: (1) an in-depth qualitative understanding of how participants used a crowd-powered cognitive reappraisal system in the wild; and (2) detailed codebooks that capture informative context about negative input thoughts and reframes. Our results surface data-derived hypotheses that may help to explain what types of reframes are helpful for users, while also providing guidance to future researchers and developers interested in building collaborative systems for mental health. In our discussion, we outline implications for systems research to leverage peer training and support, as well as opportunities to integrate AI/ML-based algorithms to support the cognitive reappraisal task. (Note: This paper includes potentially triggering mentions of mental health issues and suicide.)

[1]  B. Doré,et al.  Linguistic Synchrony Predicts the Immediate and Lasting Impact of Text-Based Emotional Support , 2018, Psychological science.

[2]  A. Greenshaw,et al.  Cross-sectional survey evaluating Text4Mood: mobile health program to reduce psychological treatment gap in mental healthcare in Alberta through daily supportive text messages , 2016, BMC Psychiatry.

[3]  Ingmar Weber,et al.  Understanding Abuse: A Typology of Abusive Language Detection Subtasks , 2017, ALW@ACL.

[4]  Xiaojuan Ma,et al.  Exploring the Effects of Technological Writing Assistance for Support Providers in Online Mental Health Community , 2020, CHI.

[5]  Judith S. Olson,et al.  Ways of Knowing in HCI , 2014, Springer New York.

[6]  Svetlana Yarosh,et al.  Video-Mediated Peer Support in an Online Community for Recovery from Substance Use Disorders , 2017, CSCW.

[7]  Munmun De Choudhury,et al.  Recovery Amid Pro-Anorexia: Analysis of Recovery in Social Media , 2016, CHI.

[8]  Michael S. Bernstein,et al.  Soylent: a word processor with a crowd inside , 2010, UIST.

[9]  John Riedl,et al.  Rating support interfaces to improve user experience and recommender accuracy , 2013, RecSys.

[10]  Aaron Halfaker,et al.  Keeping Community in the Loop: Understanding Wikipedia Stakeholder Values for Machine Learning-Based Systems , 2020, CHI.

[11]  Ran Tao,et al.  Cognitive-Behavioral Therapy. , 2008, Advances in experimental medicine and biology.

[12]  D. Mohr,et al.  Behavioral intervention technologies: evidence review and recommendations for future research in mental health. , 2013, General hospital psychiatry.

[13]  Nathan L. Vanderford,et al.  Evidence for a mental health crisis in graduate education , 2018, Nature Biotechnology.

[14]  J. Brady,et al.  The Belmont Report. Ethical principles and guidelines for the protection of human subjects of research. , 2015, The Journal of the American College of Dentists.

[15]  Loren Terveen,et al.  What is Spiritual Support and How Might It Impact the Design of Online Communities? , 2021, Proc. ACM Hum. Comput. Interact..

[16]  Eshwar Chandrasekharan,et al.  Crossmod: A Cross-Community Learning-based System to Assist Reddit Moderators , 2019, Proc. ACM Hum. Comput. Interact..

[17]  Andy Leung,et al.  Building a Digital Platform for Behavioral Intervention Technology Research and Deployment , 2020, HICSS.

[18]  R. Spitzer,et al.  The PHQ-9 , 2001, Journal of General Internal Medicine.

[19]  Anja Thieme,et al.  Understanding Client Support Strategies to Improve Clinical Outcomes in an Online Mental Health Intervention , 2020, CHI.

[20]  Sabirat Rubya,et al.  Comparing Generic and Community-Situated Crowdsourcing for Data Validation in the Context of Recovery from Substance Use Disorders , 2020, CHI.

[21]  John Torous,et al.  Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements , 2018, Evidence Based Journals.

[22]  Casey Fiesler,et al.  "We Are the Product": Public Reactions to Online Data Sharing and Privacy Controversies in the Media , 2018, CHI.

[23]  Rebecca C Rossom,et al.  IntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety , 2017, Journal of medical Internet research.

[24]  Ran Gilad-Bachrach,et al.  PopTherapy: coping with stress through pop-culture , 2014, PervasiveHealth.

[25]  Nuria Oliver,et al.  I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems , 2009, UMAP.

[26]  John Riedl,et al.  How many bits per rating? , 2012, RecSys.

[27]  I. Mauss,et al.  A Person-by-Situation Approach to Emotion Regulation , 2013, Psychological science.

[28]  R. Orji,et al.  Insights from user reviews to improve mental health apps , 2020, Health Informatics J..

[29]  Rosalind W. Picard,et al.  Crowd-powered positive psychological interventions , 2014 .

[30]  W. Robiner,et al.  The mental health professions: workforce supply and demand, issues, and challenges. , 2006, Clinical psychology review.

[31]  Christopher K. Hsee,et al.  The Evaluability Hypothesis: An Explanation for Preference Reversals between Joint and Separate Evaluations of Alternatives , 1996 .

[32]  Adam Rosenstein,et al.  Identifying the Prevalence of the Impostor Phenomenon Among Computer Science Students , 2020, SIGCSE.

[33]  Julie A Kientz,et al.  Randomized, controlled pilot trial of a smartphone app for smoking cessation using acceptance and commitment therapy. , 2014, Drug and alcohol dependence.

[34]  The mental health of PhD researchers demands urgent attention , 2019, Nature.

[35]  D. Richards Features and benefits of online counselling: Trinity College online mental health community , 2009 .

[36]  L. Timulak,et al.  Client-identified helpful and hindering events in therapist-delivered vs. self-administered online cognitive-behavioural treatments for depression in college students , 2012 .

[37]  Heleen Riper,et al.  The Pace of Technologic Change: Implications for Digital Health Behavior Intervention Research. , 2016, American journal of preventive medicine.

[38]  Charles Abraham,et al.  Mobile Phone and Web 2.0 Technologies for Weight Management: A Systematic Scoping Review , 2015, Journal of medical Internet research.

[39]  Eshwar Chandrasekharan,et al.  Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations , 2021, WWW.

[40]  Mehrbakhsh Nilashi,et al.  Collaborative filtering recommender systems , 2013 .

[41]  Robert Morris,et al.  Crowdsourcing mental health and emotional well-being , 2015 .

[42]  Casey Fiesler,et al.  Ethical Considerations for Research Involving (Speculative) Public Data , 2019, Proc. ACM Hum. Comput. Interact..

[43]  BruckmanAmy,et al.  Human-Machine Collaboration for Content Regulation , 2019 .

[44]  Rosalind W. Picard,et al.  Consensus Statement on Ethical & Safety Practices for Conducting Digital Monitoring Studies with People at Risk of Suicide and Related Behaviors , 2020, Psychiatric research and clinical practice.

[45]  Aaron Halfaker,et al.  ORES: Lowering Barriers with Participatory Machine Learning in Wikipedia , 2020, Proc. ACM Hum. Comput. Interact..

[46]  A. Kazdin,et al.  Rebooting Psychotherapy Research and Practice to Reduce the Burden of Mental Illness , 2011, Perspectives on psychological science : a journal of the Association for Psychological Science.

[47]  J. Gabrieli,et al.  Gender Differences in Emotion Regulation: An fMRI Study of Cognitive Reappraisal , 2008, Group processes & intergroup relations : GPIR.

[48]  P. Todd,et al.  Can There Ever Be Too Many Options? A Meta-Analytic Review of Choice Overload , 2010 .

[49]  James J Gross,et al.  Unpacking Cognitive Reappraisal: Goals, Tactics, and Outcomes Henderson, and Ama Thrasher for Their Help with Task Construction and Data Collection , 2022 .

[50]  Ross A. Thompson,et al.  Emotion regulation: Conceptual foundations , 2007 .

[51]  E. Phelps,et al.  Cognitive emotion regulation fails the stress test , 2013, Proceedings of the National Academy of Sciences.

[52]  Munmun De Choudhury,et al.  A Taxonomy of Ethical Tensions in Inferring Mental Health States from Social Media , 2019, FAT.

[53]  G. Petrič,et al.  Collective Empowerment in Online Health Communities: Scale Development and Empirical Validation , 2019, Journal of medical Internet research.

[54]  K. Fitzpatrick,et al.  Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial , 2017, JMIR mental health.

[55]  Mary Czerwinski,et al.  Pocket Skills: A Conversational Mobile Web App To Support Dialectical Behavioral Therapy , 2018, CHI.

[56]  Ethan Kross,et al.  Science Current Directions in Psychological Making Meaning out of Negative Experiences by Self-distancing on Behalf Of: Association for Psychological Science , 2022 .

[57]  V. Kraaij,et al.  Relationships between cognitive emotion regulation strategies and depressive symptoms: A comparative study of five specific samples , 2006 .

[58]  Bethany A. Teachman,et al.  Identification of Imminent Suicide Risk Among Young Adults using Text Messages , 2018, CHI.

[59]  David C. Atkins,et al.  Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach , 2021, WWW.

[60]  Joseph A. Konstan,et al.  Rating-Based Collaborative Filtering: Algorithms and Evaluation , 2018, Social Information Access.

[61]  Rosalind W. Picard,et al.  Helping Others Regulate Emotion Predicts Increased Regulation of One’s Own Emotions and Decreased Symptoms of Depression , 2017, Personality & social psychology bulletin.

[62]  N. Meiran,et al.  Better Late Than Never? On the Dynamics of Online Regulation of Sadness Using Distraction and Cognitive Reappraisal , 2007, Personality & social psychology bulletin.

[63]  Katie A. Siek,et al.  "Be Grateful You Don't Have a Real Disease": Understanding Rare Disease Relationships , 2017, CHI.

[64]  H. Christensen,et al.  Free range users and one hit wonders: community users of an Internet-based cognitive behaviour therapy program. , 2006, The Australian and New Zealand journal of psychiatry.

[65]  M. Zimmer “But the data is already public”: on the ethics of research in Facebook , 2010, Ethics and Information Technology.

[66]  Aniket Kittur,et al.  Can you ever trust a wiki?: impacting perceived trustworthiness in wikipedia , 2008, CSCW.

[67]  Wanda Pratt,et al.  “Suddenly, we got to become therapists for each other”: Designing Peer Support Chats for Mental Health , 2018, CHI.

[68]  Moïra Mikolajczak,et al.  Extrinsic emotion regulation. , 2020, Emotion.

[69]  D. Bravata,et al.  Prevalence, Predictors, and Treatment of Impostor Syndrome: a Systematic Review , 2019, Journal of General Internal Medicine.

[70]  R. Morris,et al.  Towards an Artificially Empathic Conversational Agent for Mental Health Applications: System Design and User Perceptions , 2018, Journal of medical Internet research.

[71]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[72]  Sean A. Munson,et al.  Design Opportunities for Mental Health Peer Support Technologies , 2017, CSCW.

[73]  J. Prescott,et al.  Online mental health communities, self-efficacy and transition to further support , 2020 .

[74]  Aaron Halfaker,et al.  Open algorithmic systems: lessons on opening the black box from Wikipedia , 2016 .

[75]  Maria Klara Wolters,et al.  The Emotional Work of Doing eHealth Research , 2017, CHI Extended Abstracts.

[76]  Rosalind W. Picard,et al.  Efficacy of a Web-Based, Crowdsourced Peer-To-Peer Cognitive Reappraisal Platform for Depression: Randomized Controlled Trial , 2015, Journal of medical Internet research.

[77]  Casey Fiesler,et al.  No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service , 2020, ICWSM.

[78]  Frank H Wilhelm,et al.  Seeing the Silver Lining: Cognitive Reappraisal Ability Moderates the Relationship between Stress and Depressive Symptoms Cognitive Reappraisal , 2022 .

[79]  Timnit Gebru,et al.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification , 2018, FAT.

[80]  J. Frost,et al.  Social Uses of Personal Health Information Within PatientsLikeMe, an Online Patient Community: What Can Happen When Patients Have Access to One Another’s Data , 2008, Journal of medical Internet research.

[81]  J. Zaki,et al.  Interpersonal emotion regulation. , 2013, Emotion.

[82]  Tristen K. Inagaki,et al.  On the Benefits of Giving Social Support , 2017 .

[83]  A. Osman,et al.  The Suicidal Behaviors Questionnaire-Revised (SBQ-R):Validation with Clinical and Nonclinical Samples , 2001, Assessment.

[84]  M. Mongrain,et al.  Acts of Kindness Reduce Depression in Individuals Low on Agreeableness , 2018, Translational Issues in Psychological Science.

[85]  J. Gross Antecedent- and response-focused emotion regulation: divergent consequences for experience, expression, and physiology. , 1998, Journal of personality and social psychology.

[86]  Svetlana Yarosh,et al.  "I Cannot Do All of This Alone": Exploring Instrumental and Prayer Support in Online Health Communities , 2020, ArXiv.

[87]  Ranjitha Kumar,et al.  Automatic retargeting of web page content , 2009, CHI Extended Abstracts.

[88]  Haiyi Zhu,et al.  Disseminating Research News in HCI: Perceived Hazards, How-To's, and Opportunities for Innovation , 2020, CHI.

[89]  M. Linehan,et al.  Effectiveness of inpatient dialectical behavioral therapy for borderline personality disorder: a controlled trial. , 2004, Behaviour research and therapy.

[90]  B. Löwe,et al.  A brief measure for assessing generalized anxiety disorder: the GAD-7. , 2006, Archives of internal medicine.

[91]  F. Crestani,et al.  A Survey of Computational Methods for Online Mental State Assessment on Social Media , 2021, ACM Trans. Comput. Heal..

[92]  Nazanin Andalibi,et al.  Considerations in Designing Digital Peer Support for Mental Health: Interview Study Among Users of a Digital Support System (Buddy Project) , 2020, JMIR mental health.

[93]  John Riedl,et al.  Is seeing believing?: how recommender system interfaces affect users' opinions , 2003, CHI '03.