Designing a virtual patient dialogue system based on terminology-rich resources: Challenges and evaluation

Virtual patient software allows health professionals to practice their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we de- signed a patient record model, a knowledge model for the task, and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowl- edge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 terms, and dictionaries with over 959,000 word/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non- medical evaluators) interact with the system and use 35 cases from 18 specialities. We con- ducted a quantitative evaluation of all components by analysing interaction logs (11,834 turns). Natural language understanding achieved an F-measure of 95.8 per cent. Dialogue management provided on average 74.3 (±9.5) per cent of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All eval- uated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8 per cent of their terms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.

[1]  Mary T. Johnson,et al.  Development of Virtual Patient Simulations for Medical Education , 2009 .

[2]  Allen C. Browne,et al.  Lexical methods for managing variation in biomedical terminologies. , 1994, Proceedings. Symposium on Computer Applications in Medical Care.

[3]  Juan Luis Castro,et al.  A case based reasoning model for multilingual language generation in dialogues , 2012, Expert Syst. Appl..

[4]  Albert A. Rizzo,et al.  Sorting Out the Virtual Patient: How to Exploit Artificial Intelligence, Game Technology and Sound Educational Practices to Create Engaging Role-Playing Simulations , 2012, Int. J. Gaming Comput. Mediat. Simulations.

[5]  Peter L. Elkin,et al.  UMLS Concept Indexing for Production Databases: A Feasibility Study , 2001, J. Am. Medical Informatics Assoc..

[6]  Timothy Bickmore,et al.  Health dialog systems for patients and consumers , 2006, J. Biomed. Informatics.

[7]  P N Kizakevich,et al.  The virtual standardized patient. Simulated patient-practitioner dialog for patient interview training. , 2000, Studies in health technology and informatics.

[8]  Marilyn A. Walker,et al.  PARADISE: A Framework for Evaluating Spoken Dialogue Agents , 1997, ACL.

[9]  Kyle Johnsen,et al.  Evaluating a Script -Based Approach for Simulating Patient -Doctor Interaction , 2005 .

[10]  Olivier Galibert,et al.  The LIMSI Participation to the QAst Track , 2007, CLEF.

[11]  Min Li,et al.  An ontology for clinical questions about the contents of patient notes , 2012, J. Biomed. Informatics.

[12]  Rieks op den Akker,et al.  Handling speech input in the ritel QA dialogue system , 2007, INTERSPEECH.

[13]  Ralph Weischedel,et al.  PERFORMANCE MEASURES FOR INFORMATION EXTRACTION , 2007 .

[14]  Albert A. Rizzo,et al.  Evaluation of Justina: A Virtual Patient with PTSD , 2008, IVA.

[15]  Pierre Zweigenbaum,et al.  Description of the PatientGenesys Dialogue System , 2015, SIGDIAL Conference.

[16]  Robert H. Baud,et al.  UMLF: a Unified Medical Lexicon for French , 2005, AMIA.

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Evan Jaffe,et al.  A Corpus of Word-Aligned Asked and Anticipated Questions in a Virtual Patient Dialogue System , 2016, LREC.

[19]  Sergei Nirenburg,et al.  A Unified Ontological-Semantic Substrate for Physiological Simulation and Cognitive Modeling , 2015 .

[20]  Félicie Pastore.  How Can I Help You Today ? Guide de la Consultation Médicale et Paramédicale en Anglais , 2016 .

[21]  Evan Jaffe,et al.  Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System , 2017, BEA@EMNLP.

[22]  Dilek Z. Hakkani-Tür,et al.  Deep Learning for Spoken and Text Dialog Systems , 2017 .

[23]  David R. Traum,et al.  Evaluation of Multi-party Virtual Reality Dialogue Interaction , 2004, LREC.

[24]  D. Cook,et al.  Computerized Virtual Patients in Health Professions Education: A Systematic Review and Meta-Analysis , 2010, Academic medicine : journal of the Association of American Medical Colleges.

[25]  Stephanie Seneff,et al.  Spoken Dialogue Systems , 2008 .

[26]  Albert Rizzo,et al.  Designing Useful Virtual Standardized Patient Encounters , 2012 .

[27]  Sergei Nirenburg,et al.  Language Understanding in Maryland Virtual Patient , 2008, SPSCTPA@COLING.

[28]  Jonathan Ginzburg,et al.  On the Means for Clarification in Dialogue , 2001, SIGDIAL Workshop.

[29]  Kristy Oden,et al.  Virtual Standardized Patient , 2018, International Journal for Innovation Education and Research.

[30]  Tim Paek.  Empirical Methods for Evaluating Dialog Systems , 2001, SIGDIAL Workshop.

[31]  Douglas Danforth,et al.  Developing a Conversational Virtual Standardized Patient to Enable Students to Practice History-Taking Skills , 2017, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[32]  Patrick G. Kenny,et al.  Embodied Conversational Virtual Patients , 2010 .

[33]  Florian Pinault.  Apprentissage par renforcement pour la généralisation des approches automatiques dans la conception des systèmes de dialogue oral. (Statistical methods for a oral human-machine dialog system) , 2011 .

[34]  John Fox,et al.  Automatic generation of spoken dialogue from medical plans and ontologies , 2006, J. Biomed. Informatics.

[35]  Paul N Kizakevich,et al.  Lessons learned in modeling virtual pediatric patients. , 2003, Studies in health technology and informatics.

[36]  Pierre Zweigenbaum,et al.  Managing Linguistic and Terminological Variation in a Medical Dialogue System , 2016, LREC.

[37]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[38]  Steve J. Young,et al.  USING POMDPS FOR DIALOG MANAGEMENT , 2006, 2006 IEEE Spoken Language Technology Workshop.

[39]  Brent Rossen,et al.  Human-Centered Distributed Conversational Modeling: Efficient Modeling of Robust Virtual Human Conversations , 2009, IVA.

[40]  Toni Giorgino,et al.  Automated Spoken Dialog System for Hypertensive Patient Home Management , 2004 .

[41]  Jan Svartvik,et al.  A __ comprehensive grammar of the English language , 1988 .

[42]  Amy O. Stevens,et al.  The use of virtual patients to teach medical students history taking and communication skills. , 2006, American journal of surgery.

[43]  Chunhua Weng,et al.  Leveraging dialog systems research to assist biomedical researchers' interrogation of Big Clinical Data , 2016, J. Biomed. Informatics.

[44]  Laura E. Barnes,et al.  A Deep Learning Methodology for Semantic Utterance Classification in Virtual Human Dialogue Systems , 2016, IVA.

[45]  Nabil Hathout,et al.  An Experimental Constructional Database : The MorTAL Project , 2002 .

[46]  Olivier Galibert,et al.  Integrating spoken dialog and question answering: the ritel project , 2006, INTERSPEECH.

[47]  Pierre Zweigenbaum,et al.  Acquiring meaning for French medical terminology: contribution of morphosemantics , 2004, MedInfo.

[48]  Brent Rossen,et al.  A crowdsourcing method to develop virtual human conversational agents , 2012, Int. J. Hum. Comput. Stud..

[49]  Neal Benedict,et al.  TEACHERS' TOPICS Virtual Patients and Problem-Based Learning in Advanced Therapeutics , 2010 .

[50]  Ronald A. Cole,et al.  TOOLS FOR RESEARCH AND EDUCATION IN SPEECH SCIENCE , 1999 .

[51]  Gale M. Lucas,et al.  Natural Language Understanding Performance & Use Considerations in Virtual Medical Encounters , 2016, MMVR.

[52]  Timothy W. Bickmore.  Conversational Agents for Automated Inpatient and Outpatient Health Counseling , 2015, AMIA.

[53]  Olivier Bodenreider,et al.  Aggregating UMLS Semantic Types for Reducing Conceptual Complexity , 2001, MedInfo.

[54]  Banani Roy,et al.  Methods for Evaluating Software Architecture: A Survey , 2008 .

[55]  Michael F. McTear,et al.  Handling errors and determining confirmation strategies - An object-based approach , 2003, Speech Commun..

[56]  Pierre Zweigenbaum,et al.  Transfer-Based Learning-to-Rank Assessment of Medical Term Technicality , 2016, LREC.

[57]  M. McShane,et al.  ADAPTIVITY IN A MULTI-AGENT CLINICAL SIMULATION SYSTEM , 2008 .

[58]  B. Courtois,et al.  Un système de dictionnaires électroniques pour les mots simples du français , 1990 .

[59]  Pierre Zweigenbaum,et al.  Automatic classification of doctor-patient questions for a virtual patient record query task , 2017, BioNLP.

[60]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..

[61]  Kevin Donnelly,et al.  SNOMED-CT: The advanced terminology and coding system for eHealth. , 2006, Studies in health technology and informatics.

[62]  David Traum,et al.  The Information State Approach to Dialogue Management , 2003 .