Content-Based Recommender Support System for Counselors in a Suicide Prevention Chat Helpline: Design and Evaluation Study

Background The working environment of a suicide prevention helpline requires high emotional and cognitive awareness from chat counselors. A shared opinion among counselors is that as a chat conversation becomes more difficult, it takes more effort and a longer amount of time to compose a response, which, in turn, can lead to writer’s block. Objective This study evaluates and then designs supportive technology to determine if a support system that provides inspiration can help counselors resolve writer’s block when they encounter difficult situations in chats with help-seekers. Methods A content-based recommender system with sentence embedding was used to search a chat corpus for similar chat situations. The system showed a counselor the most similar parts of former chat conversations so that the counselor would be able to use approaches previously taken by their colleagues as inspiration. In a within-subject experiment, counselors’ chat replies when confronted with a difficult situation were analyzed to determine if experts could see a noticeable difference in chat replies that were obtained in 3 conditions: (1) with the help of the support system, (2) with written advice from a senior counselor, or (3) when receiving no help. In addition, the system’s utility and usability were measured, and the validity of the algorithm was examined. Results A total of 24 counselors used a prototype of the support system; the results showed that, by reading chat replies, experts were able to significantly predict if counselors had received help from the support system or from a senior counselor (P=.004). Counselors scored the information they received from a senior counselor (M=1.46, SD 1.91) as significantly more helpful than the information received from the support system or when no help was given at all (M=–0.21, SD 2.26). Finally, compared with randomly selected former chat conversations, counselors rated the ones identified by the content-based recommendation system as significantly more similar to their current chats (β=.30, P<.001). Conclusions Support given to counselors influenced how they responded in difficult conversations. However, the higher utility scores given for the advice from senior counselors seem to indicate that specific actionable instructions are preferred. We expect that these findings will be beneficial for developing a system that can use similar chat situations to generate advice in a descriptive style, hence helping counselors through writer’s block.

[1]  Timothy L. Acorn,et al.  Smart: Support: Management Automated Reasoning Technology for Compaq Customer Service , 1992, IAAI.

[2]  R. Cialdini.  Influence: The Psychology of Persuasion , 1993 .

[3]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[4]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[5]  Clifford Nass,et al.  Helper agent: designing an assistant for human-human interaction in a virtual meeting space , 2000, CHI.

[6]  Tsuhan Chen,et al.  An active learning framework for content-based information retrieval , 2002, IEEE Trans. Multim..

[7]  H. Abdi.  The Bonferonni and Šidák Corrections for Multiple Comparisons , 2006 .

[8]  John Kalafat,et al.  An evaluation of crisis hotline outcomes. Part 2: Suicidal callers. , 2007, Suicide & life-threatening behavior.

[9]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[10]  Mari Carmen Puerta Melguizo,et al.  The effects of the number of links and navigation support on cognitive load and learning with hypertext: The mediating role of reading order , 2009, Comput. Hum. Behav..

[11]  Jo M. A. Hermanns,et al.  Counseling children at a helpline: chatting or calling? , 2009 .

[12]  J. Suler.  The Psychology of Text Relationships , 2011 .

[13]  George Angelos Papadopoulos,et al.  A Context Aware Recommender System for Creativity Support Tools , 2011, J. Univers. Comput. Sci..

[14]  Lorie L. Sicafuse,et al.  SOMEONE TO LISTEN: INCREASING YOUTH HELP-SEEKING BEHAVIOR THROUGH A TEXT-BASED CRISIS LINE FOR YOUTH , 2013 .

[15]  M. Pratt,et al.  The Future of Volunteers in Crisis Hotline Work , 2013 .

[16]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[17]  Debra Rickwood,et al.  Online Counseling and Therapy for Mental Health Problems: A Systematic Review of Individual Synchronous Interventions Using Chat , 2013 .

[18]  Georgiana Dinu,et al.  Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors , 2014, ACL.

[19]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.

[20]  Oded Nov,et al.  Influencing Retirement Saving Behavior with Expert Advice and Social Comparison as Persuasive Techniques , 2015, PERSUASIVE.

[21]  Henry Lieberman,et al.  Mixed-Initiative Real-Time Topic Modeling & Visualization for Crisis Counseling , 2015, IUI.

[22]  Konrad P. Körding,et al.  Science Concierge: A Fast Content-Based Recommendation System for Scientific Publications , 2016, PloS one.

[23]  Jure Leskovec,et al.  Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health , 2016, TACL.

[24]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[25]  Sanjeev Arora,et al.  A Simple but Tough-to-Beat Baseline for Sentence Embeddings , 2017, ICLR.

[26]  Wei Chu,et al.  AliMe Chat: A Sequence to Sequence and Rerank based Chatbot Engine , 2017, ACL.

[27]  J. Smit,et al.  Evaluation of the 113Online Suicide Prevention Crisis Chat Service: Outcomes, Helper Behaviors and Comparison to Telephone Hotlines , 2017, Suicide & life-threatening behavior.

[28]  Colin G. Walsh,et al.  Predicting Risk of Suicide Attempts Over Time Through Machine Learning , 2017 .

[29]  Ray Kurzweil,et al.  Learning Semantic Textual Similarity from Conversations , 2018, Rep4NLP@ACL.

[30]  Alex B. Fine,et al.  Natural Language Processing of Social Media as Screening for Suicide Risk , 2018, Biomedical informatics insights.

[31]  Honain Mohib Derrar,et al.  Clustering for the automatic annotation of customer service chat messages , 2019 .

[32]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.