Multilingual Conversational Systems to Drive the Collection of Patient-Reported Outcomes and Integration into Clinical Workflows

Patient-reported outcomes (PROs) and their use in the clinical workflow can improve cancer survivors’ outcomes and quality of life. However, there are several challenges regarding efficient collection of the patient-reported outcomes and their integration into the clinical workflow. Patient adherence and interoperability are recognized as main barriers. This work implements a cancer-related study which interconnects artificial intelligence (spoken language algorithms, conversational intelligence) and natural sciences (embodied conversational agents) to create an omni-comprehensive system enabling symmetric computer-mediated interaction. Its goal is to collect patient information and integrate it into clinical routine as digital patient resources (the Fast Healthcare Interoperability Resources). To further increase convenience and simplicity of the data collection, a multimodal sensing network is delivered. In this paper, we introduce the main components of the system, including the mHealth application, the Open Health Connect platform, and algorithms to deliver speech enabled 3D embodied conversational agent to interact with the cancer survivors in five different languages. The system integrates cancer patients’ reported information as patient gathered health data into their digital clinical record. The value and impact of the integration will be further evaluated in the clinical study.

[1]  Cinzia Brunelli,et al.  Bmc Medical Informatics and Decision Making a Wireless Health Outcomes Monitoring System (whoms): Development and Field Testing with Cancer Patients Using Mobile Phones , 2022 .

[2]  C. LeRouge,et al.  Capturing the patient voice: implementing patient-reported outcomes across the health system , 2019, Quality of Life Research.

[3]  Morgan C. Fitzgerald,et al.  Improving patient engagement, adherence, and satisfaction in lung cancer surgery with implementation of a mobile device platform for patient reported outcomes , 2020, Journal of thoracic disease.

[4]  Hermie Hermens,et al.  Design Features of Embodied Conversational Agents in eHealth: a Literature Review , 2020, Int. J. Hum. Comput. Stud..

[5]  Nelson Pacheco da Rocha,et al.  Ambient Assisted Living and Health-Related Outcomes - A Systematic Literature Review , 2017, Informatics.

[6]  Ho Ming Lau,et al.  Embodied Conversational Agents in Clinical Psychology: A Scoping Review , 2017, Journal of medical Internet research.

[7]  Joseph Weizenbaum,et al.  ELIZA—a computer program for the study of natural language communication between man and machine , 1966, CACM.

[8]  Nick Pawlowski,et al.  Rasa: Open Source Language Understanding and Dialogue Management , 2017, ArXiv.

[9]  S. Greenfield,et al.  The Medical Outcomes Study. An application of methods for monitoring the results of medical care. , 1989, JAMA.

[10]  Randy Klaassen,et al.  Design and Evaluation of a Pervasive Coaching and Gamification Platform for Young Diabetes Patients † , 2018, Sensors.

[11]  Otis L. Owens,et al.  Evaluation of a Computer-Based Decision Aid for Promoting Informed Prostate Cancer Screening Decisions Among African American Men: iDecide , 2018, American journal of health promotion : AJHP.

[12]  Jessica A. Chen,et al.  Conversational agents in healthcare: a systematic review , 2018, J. Am. Medical Informatics Assoc..

[13]  T. Okba,et al.  A mammographic images classification technique via the Gaussian Radial Basis Kernel ELM and KPCA , 2021 .

[14]  Yoshua Bengio,et al.  End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results , 2014, ArXiv.

[15]  Hayfa Zgaya,et al.  Workflow tool to Model and simulate patients paths in Pediatric Emergency Department , 2013 .

[16]  Maite López-Sánchez,et al.  Enhancing sentient embodied conversational agents with machine learning , 2020, Pattern Recognit. Lett..

[17]  Mohamed Hamed N. Taha,et al.  Breast and Colon Cancer Classification from Gene Expression Profiles Using Data Mining Techniques , 2020, Symmetry.

[18]  G. Liao,et al.  Artificial intelligence performance in detecting tumor metastasis from medical radiology imaging: A systematic review and meta-analysis , 2020, EClinicalMedicine.

[19]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[20]  Abhishek Singh,et al.  Introduction to Microsoft Bot, RASA, and Google Dialogflow , 2019, Building an Enterprise Chatbot.

[21]  Anthony Arnold,et al.  eHealth System for Collecting and Utilizing Patient Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care) Among Cancer Patients: Mixed Methods Approach to Evaluate Feasibility and Acceptability , 2017, Journal of medical Internet research.

[22]  Alexander Todorov,et al.  What Is Typical Is Good , 2015, Psychological science.

[23]  Michael Schoeffler,et al.  webMUSHRA — A Comprehensive Framework for Web-based Listening Tests , 2018 .

[24]  Anke Oenema,et al.  Motivational Interviewing in a Web-Based Physical Activity Intervention With an Avatar: Randomized Controlled Trial , 2014, Journal of medical Internet research.

[25]  Kien Hoa Ly,et al.  A fully automated conversational agent for promoting mental well-being: A pilot RCT using mixed methods , 2017, Internet interventions.

[26]  M. Dimatteo,et al.  The challenge of patient adherence , 2005, Therapeutics and clinical risk management.

[27]  Imran Razzak,et al.  Leveraging big data analytics in healthcare enhancement: trends, challenges and opportunities , 2020, Multimedia Systems.

[28]  Peter A. Gloor,et al.  In the shades of the uncanny valley: An experimental study of human-chatbot interaction , 2018, Future Gener. Comput. Syst..

[29]  Raees Ahmad Khan,et al.  Evaluating the Impact of Different Symmetrical Models of Ambient Assisted Living Systems , 2021, Symmetry.

[30]  Lex van Velsen,et al.  Developing Embodied Conversational Agents for Coaching People in a Healthy Lifestyle: Scoping Review , 2020, Journal of medical Internet research.

[31]  K. Mandl,et al.  SMART Markers: collecting patient-generated health data as a standardized property of health information technology. , 2020, NPJ digital medicine.

[32]  Boris Ginsburg,et al.  Jasper: An End-to-End Convolutional Neural Acoustic Model , 2019, INTERSPEECH.

[33]  K. Fitzpatrick,et al.  Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial , 2017, JMIR mental health.

[34]  B. Leiby,et al.  Utilizing Digital Health to Collect Electronic Patient-Reported Outcomes in Prostate Cancer: Single-Arm Pilot Trial , 2020, Journal of medical Internet research.

[35]  Johannes Ettl,et al.  PROutine: a feasibility study assessing surveillance of electronic patient reported outcomes and adherence via smartphone app in advanced cancer. , 2017, Annals of palliative medicine.

[36]  Timothy W Bickmore,et al.  Engaging women with an embodied conversational agent to deliver mindfulness and lifestyle recommendations: A feasibility randomized control trial. , 2017, Patient education and counseling.

[37]  B. Inkster,et al.  An Empathy-Driven, Conversational Artificial Intelligence Agent (Wysa) for Digital Mental Well-Being: Real-World Data Evaluation Mixed-Methods Study , 2018, JMIR mHealth and uHealth.

[38]  Rakesh Kumar Sharma.  An Analytical Study and Review of open source Chatbot framework, Rasa , 2020 .

[39]  Jean-Emmanuel Bibault,et al.  Healthcare ex Machina: Are conversational agents ready for prime time in oncology? , 2019, Clinical and translational radiation oncology.

[40]  Naomi C. Brownstein,et al.  Innovations in research and clinical care using patient‐generated health data , 2020, CA: a cancer journal for clinicians.

[41]  C. Sidner,et al.  Automated interventions for multiple health behaviors using conversational agents. , 2013, Patient education and counseling.

[42]  Domenico Conforti,et al.  Multi-Classifier Approaches for Supporting Clinical Decision Making , 2020, Symmetry.

[43]  T. Bickmore,et al.  Using Relational Agents to Promote Exercise and Sun Protection: Assessment of Participants’ Experiences With Two Interventions , 2018, Journal of medical Internet research.

[44]  Eric Atwell,et al.  Chatbots: Are they Really Useful? , 2007, LDV Forum.

[45]  L. Koppert,et al.  Machine learning with PROs in breast cancer surgery; caution: Collecting PROs at baseline is crucial , 2020, The breast journal.

[46]  Matej Rojc,et al.  The TTS-driven affective embodied conversational agent EVA, based on a novel conversational-behavior generation algorithm , 2017, Eng. Appl. Artif. Intell..

[47]  Rosalind W. Picard,et al.  Establishing the computer-patient working alliance in automated health behavior change interventions. , 2005, Patient education and counseling.

[48]  Z. Kacic,et al.  Advanced Content and Interface Personalization through Conversational Behavior and Affective Embodied Conversational Agents , 2018, Artificial Intelligence - Emerging Trends and Applications.

[49]  Yan Fossat,et al.  Physicians’ Perceptions of Chatbots in Health Care: Cross-Sectional Web-Based Survey , 2018, Journal of medical Internet research.

[50]  Pooja Rajguru,et al.  A novel method for collecting patient reported outcomes (PROs): developing and validating electronic PROs on a mobile smartphone platform , 2020 .

[51]  Timothy W. Bickmore,et al.  'It's just like you talk to a friend' relational agents for older adults , 2005, Interact. Comput..