From data to text in the Neonatal Intensive Care Unit: Using NLG technology for decision support and information management

Contemporary Neonatal Intensive Care Units collect vast amounts of patient data in various formats, making efficient processing of information by medical professionals difficult. Moreover, different stakeholders in the neonatal scenario, which include parents as well as staff occupying different roles, have different information requirements. This paper describes recent and ongoing work on building systems that automatically generate textual summaries of neonatal data. Our evaluation results show that the technology is viable and comparable in its effectiveness for decision support to existing presentation modalities. We discuss the lessons learned so far, as well as the major challenges involved in extending current technology to deal with a broader range of data types, and to improve the textual output in the form of more coherent summaries.

[1]  Wolfgang Hoeppner,et al.  Review of Generating natural language under pragmantic constraints by Edward H. Hovy. Lawrence Erlbaum Associates 1988. , 1990 .

[2]  Floriana Grasso,et al.  Affective Natural Language Generation , 1999, IWAI.

[3]  Evert de Jonge,et al.  An empirical comparison of four procedures for filtering monitoring data , 2007 .

[4]  Mimo Caenepeel Aspect and text structure , 1995 .

[5]  M. Chambrin Alarms in the intensive care unit: how can the number of false alarms be reduced? , 2001, Critical care.

[6]  Albert Gatt,et al.  The Importance of Narrative and Other Lessons from an Evaluation of an NLG System that Summarises Clinical Data , 2008, INLG.

[7]  L. Elting,et al.  Is a Picture Worth a Thousand Medical Words? A Randomized Trial of Reporting Formats for Medical Research Data , 1991, Methods of Information in Medicine.

[8]  Michael White,et al.  Towards broad coverage surface realization with CCG , 2007, MTSUMMIT.

[9]  Carlota S. Smith,et al.  The Domain of Tense , 2009 .

[10]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[11]  Fleur T. Tehrani,et al.  Intelligent decision support systems for mechanical ventilation , 2008, Artif. Intell. Medicine.

[12]  George Hripcsak,et al.  Temporal reasoning with medical data - A review with emphasis on medical natural language processing , 2007, J. Biomed. Informatics.

[13]  Angus Roberts,et al.  Mining clinical relationships from patient narratives , 2008, BMC Bioinformatics.

[14]  Feng Gao,et al.  Using Temporal Constraints to Integrate Signal Analysis and Domain Knowledge in Medical Event Detection , 2009, AIME.

[15]  Mark Steedman,et al.  Temporal Ontology and Temporal Reference , 1988, CL.

[16]  Alison Cawsey,et al.  The Evaluation of a Personalised Health Information System for Patients with Cancer , 2000, User Modeling and User-Adapted Interaction.

[17]  Avi Parush,et al.  Helping People with Visual Impairments Gain Access to Graphical Information Through Natural Language: The iGraph System , 2006, ICCHP.

[18]  Federico Peinado,et al.  Narrative Models : Narratology Meets Artificial Intelligence , 2006 .

[19]  Rolf A. Zwaan,et al.  TIME IN NARRATIVE COMPREHENSION: A COGNITIVE PERSPECTIVE , 2009 .

[20]  Eduard Hovy,et al.  Generating Natural Language Under Pragmatic Constraints , 1988 .

[21]  Hans Reichenbach,et al.  Elements of symbolic logic , 1948 .

[22]  Frank Puppe,et al.  Application and Evaluation of a Medical Knowledge System in Sonography (SONOCONSULT) , 2008, ECAI.

[23]  S Miksch,et al.  Effective data validation of high-frequency data: Time-point-, time-interval-, and trend-based methods , 1997, Comput. Biol. Medicine.

[24]  Heidrun Schumann,et al.  Visualization for modeling and simulation: visualization methods for time-dependent data - an overview , 2003, WSC '03.

[25]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[26]  Mary Dee Harris,et al.  Building a Large-scale Commercial NLG System for an EMR , 2008, INLG.

[27]  Hua Min,et al.  Consistency across the hierarchies of the UMLS Semantic Network and Metathesaurus , 2003, J. Biomed. Informatics.

[28]  Rolf A. Zwaan,et al.  Situation models in language comprehension and memory. , 1998, Psychological bulletin.

[29]  Kees van Deemter,et al.  Information sharing : reference and presupposition in language generation and interpretation , 2002 .

[30]  Jim Hunter,et al.  Automatic Generation of Textual Summaries from Neonatal Intensive Care Data , 2007, AIME.

[31]  Dov M. Gabbay,et al.  Handbook of Temporal Reasoning in Artificial Intelligence , 2005, Handbook of Temporal Reasoning in Artificial Intelligence.

[32]  H. Mcdonald,et al.  Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. , 2005, JAMA.

[33]  Yuval Shahar,et al.  Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data , 2008, Artif. Intell. Medicine.

[34]  Panagiotis Stamatopoulos,et al.  Summarization from Medical Documents: A Survey , 2005, Artif. Intell. Medicine.

[35]  Teun A. van Dijk,et al.  Discourse as structure and process , 1997 .

[36]  O. M.O.D ILEX: An Architecture for a Dynamic Hypertext Generation System , 2001 .

[37]  Richard T. Snodgrass,et al.  Extending Temporal Databases to Deal with Telic / Atelic Medical Data , 2006 .

[38]  M. Varner,et al.  A measurement model of perinatal stressors: identifying risk for postnatal emotional distress in mothers of high-risk infants. , 2000, Journal of clinical psychology.

[39]  Tsvi Kuflik,et al.  Adaptive, intelligent presentation of information for the museum visitor in PEACH , 2007, User Modeling and User-Adapted Interaction.

[40]  K. Tan,et al.  Clinical decision support systems for neonatal care. , 2005, The Cochrane database of systematic reviews.

[41]  Carolyn McGregor,et al.  Temporal abstraction in intelligent clinical data analysis: A survey , 2007, Artif. Intell. Medicine.

[42]  Alex Lascarides,et al.  Preventing False Temporal Implicatures: Interactive Defaults for Text Generation , 1992, COLING.

[43]  Chris Mellish,et al.  ILEX: an architecture for a dynamic hypertext generation system , 2001, Nat. Lang. Eng..

[44]  BentalDiana,et al.  Personalized and adaptive systems for medical consumer applications , 2002 .

[45]  Brigitte Krenn,et al.  Fully generated scripted dialogue for embodied agents , 2008, Artif. Intell..

[46]  Cécile Paris,et al.  Tailoring Object Descriptions to a User's Level of Expertise , 1988, Comput. Linguistics.

[47]  Ehud Reiter,et al.  Selecting the Content of Textual Descriptions of Geographically Located Events in Spatio-Temporal Weather Data , 2007, SGAI Conf..

[48]  Ben Shneiderman,et al.  Interactive pattern search in time series , 2005, IS&T/SPIE Electronic Imaging.

[49]  Albert Gatt,et al.  SimpleNLG: A Realisation Engine for Practical Applications , 2009, ENLG.

[50]  D Hüske-Kraus,et al.  Text Generation in Clinical Medicine – a Review , 2003, Methods of Information in Medicine.

[51]  Luca Chittaro,et al.  Information visualization and its application to medicine , 2001, Artif. Intell. Medicine.

[52]  Ielka van der Sluis,et al.  Towards Empirical Evaluation of Affective Tactical NLG , 2009, ENLG.

[53]  E. André The Generation of Multimedia Presentations , 2000 .

[54]  Richard Power,et al.  An integrated framework for text planning and pronominalisation , 2000, INLG.

[55]  Amy Isard,et al.  Multi-lingual Evaluation of a Natural Language Generation System , 2004, LREC.

[56]  Charles G. Martin,et al.  Influence of data display formats on physician investigators' decisions to stop clinical trials: prospective trial with repeated measures , 1999, BMJ.

[57]  Rebecca J. Passonneau,et al.  Situations and Intervals , 1987, ACL.

[58]  Judith Masthoff,et al.  What Do You Want to Know? Investigating the Information Requirements of Patient Supporters , 2008, 2008 21st IEEE International Symposium on Computer-Based Medical Systems.

[59]  François Pachet,et al.  NéoGanesh: a working system for the automated control of assisted ventilation in ICUs , 1997, Artif. Intell. Medicine.

[60]  Jim Hunter,et al.  Choosing words in computer-generated weather forecasts , 2005, Artif. Intell..

[61]  David Madigan,et al.  Probabilistic Temporal Reasoning , 2005, Handbook of Temporal Reasoning in Artificial Intelligence.

[62]  Gary Ewing,et al.  Role and experience determine decision support interface requirements in a neonatal intensive care environment , 2003, J. Biomed. Informatics.

[63]  T. Trabasso,et al.  Constructing inferences during narrative text comprehension. , 1994 .

[64]  Eamonn J. Keogh,et al.  An online algorithm for segmenting time series , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[65]  Michael Elhadad,et al.  An Overview of SURGE: a Reusable Comprehensive Syntactic Realization Component , 1996, INLG.

[66]  Kathleen F. McCoy,et al.  Taking Time to Structure Discourse: Pronoun Generation Beyond Accessibility , 2020, Proceedings of the Twenty First Annual Conference of the Cognitive Science Society.

[67]  Robert Logie,et al.  Expertise and the interpretation of computerized physiological data: implications for the design of computerized monitoring in neonatal intensive care , 2002, Int. J. Hum. Comput. Stud..

[68]  Luca Chittaro,et al.  Temporal Granularity and Indeterminacy in Reasoning About Actions and Change: An Approach Based on the Event Calculus , 2002, Annals of Mathematics and Artificial Intelligence.

[69]  Ben Shneiderman,et al.  Maintaining concentration to achieve task completion , 2005, DUX '05.

[70]  Wolfgang Klein,et al.  Time in language , 1994 .

[71]  B. Penninx,et al.  A Randomized Controlled Trial of a Tailored Primary Care Program to Reverse Excessive Alcohol Consumption , 2012, The Journal of the American Board of Family Medicine.

[72]  Gilhooley,et al.  The cognitive processes of doctors and nurses in the interpretation of physiological monitoring data in the neonate , 2000, Early human development.

[73]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[74]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[75]  Ralf Klabunde,et al.  Lexical choice of modal expressions , 2007, ENLG.

[76]  Ehud Reiter,et al.  SumTime-Mousam: Configurable marine weather forecast generator , 2003 .

[77]  Albert Gatt,et al.  Towards a Possibility-Theoretic Approach to Uncertainty in Medical Data Interpretation for Text Generation , 2009, KR4HC.

[78]  Alison Cawsey,et al.  Personalized and adaptive systems for medical consumer applications , 2002, CACM.

[79]  Emiel Krahmer,et al.  Efficient context-sensitive generation of referring expressions , 2002 .

[80]  Bernadette Strople,et al.  Can technology improve intershift report? What the research reveals. , 2006, Journal of professional nursing : official journal of the American Association of Colleges of Nursing.

[81]  Wolfgang Jank,et al.  Representing Unevenly-Spaced Time Series Data for Visualization and Interactive Exploration , 2005, INTERACT.

[82]  Ehud Reiter,et al.  Lessons from a failure: Generating tailored smoking cessation letters , 2003, Artif. Intell..

[83]  R. Power,et al.  Summarisation and Visualisation of e-Health Data Repositories , 2005 .

[84]  Zeno Vendler,et al.  Verbs and Times , 1957, The Language of Time - A Reader.

[85]  Roque Marín,et al.  Fuzzy theory approach for temporal model-based diagnosis: An application to medical domains , 2006, Artif. Intell. Medicine.

[86]  Dirk Kraus,et al.  Suregen-2: a shell system for the generation of clinical documents , 2003, EACL.

[87]  Vladimir Ryabov,et al.  Probabilistic temporal interval networks , 2004, Proceedings. 11th International Symposium on Temporal Representation and Reasoning, 2004. TIME 2004..

[88]  Karen Kukich,et al.  Design of a Knowledge-Based Report Generator , 1983, ACL.

[89]  Ehud Reiter,et al.  An Architecture for Data-to-Text Systems , 2007, ENLG.

[90]  B L Humphreys,et al.  The UMLS project: making the conceptual connection between users and the information they need. , 1993, Bulletin of the Medical Library Association.

[91]  R. Logie,et al.  When a graph is poorer than 100 words: A comparison of computerised natural language generation, human generated descriptions and graphical displays in neonatal intensive care , 2010 .

[92]  Judith Masthoff,et al.  Generating Referring Expressions: Making Referents Easy to Identify , 2007, Computational Linguistics.

[93]  Kees van Deemter Generating Referring Expressions that Involve Gradable Properties , 2006, CL.

[94]  Richard I. Kittredge,et al.  Using natural-language processing to produce weather forecasts , 1994, IEEE Expert.

[95]  Albert Gatt,et al.  Text Content and Task Performance in the Evaluation of a Natural Language Generation System , 2009, RANLP.

[96]  Norman Alm,et al.  Using NLG to Help Language-Impaired Users Tell Stories and Participate in Social Dialogues , 2009, ENLG.

[97]  Rolf A. Zwaan Time in Language, Situation Models, and Mental Simulations , 2008 .

[98]  William Winn,et al.  Chapter 1 Contributions of Perceptual and Cognitive Processes to the Comprehension of Graphics , 1994 .

[99]  Bonnie L. Webber,et al.  The Interpretation of Tense in Discourse , 1987, ACL.

[100]  N. McIntosh,et al.  A randomized, controlled trial of computerized physiologic trend monitoring in an intensive care unit. , 1998, Critical care medicine.

[101]  Robin I. M. Dunbar,et al.  Social networks, support cliques, and kinship , 1995, Human nature.

[102]  Anna S. Law,et al.  A Comparison of Graphical and Textual Presentations of Time Series Data to Support Medical Decision Making in the Neonatal Intensive Care Unit , 2005, Journal of Clinical Monitoring and Computing.

[103]  John A. Quinn,et al.  Factorial Switching Linear Dynamical Systems Applied to Physiological Condition Monitoring , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[104]  Didier Dubois,et al.  Fuzziness and Uncertainty in Temporal Reasoning , 2003, J. Univers. Comput. Sci..

[105]  José Coch System Demonstration Interactive Generation And Knowledge Administration In Multimeteo , 1998, INLG.

[106]  Floriana Grasso,et al.  Refining instructional text generation after evaluation , 1999, Artif. Intell. Medicine.

[107]  Gary Ewing,et al.  The NEONATE Database , 2003 .

[108]  Bruce G. Buchanan,et al.  The MYCIN Experiments of the Stanford Heuristic Programming Project , 1985 .

[109]  Janet Marie Hitzeman Temporal adverbials and the syntax-semantics interface , 1994 .

[110]  Kathleen McKeown,et al.  Tailoring Lexical Choice to the User's Vocabulary in Multimedia Explanation Generation , 1993, ACL.

[111]  Massimiliano Giacomin,et al.  The algebra IAfuz: a framework for qualitative fuzzy temporal reasoning , 2006, Artif. Intell..

[112]  James Pustejovsky,et al.  Machine Learning of Temporal Relations , 2006, ACL.

[113]  Samson W. Tu,et al.  Protégé-2000: An Open-Source Ontology-Development and Knowledge-Acquisition Environment: AMIA 2003 Open Source Expo , 2003, AMIA.

[114]  Jim Hunter,et al.  TSNet - A Distributed Architecture for Time Series Analysis , 2008, Computer-based Medical Guidelines and Protocols.

[115]  Edward H. Shortliffe,et al.  Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (The Addison-Wesley series in artificial intelligence) , 1984 .

[116]  Bernard Comrie,et al.  Time in language. , 2000 .

[117]  Rainer Bromme,et al.  Choice of Words in Doctor–Patient Communication: An Analysis of Health-Related Internet Sites , 2007, Health communication.

[118]  Anja Belz,et al.  ITRI-03-21 And Now with Feeling : Developments in Emotional Language Generation , 2004 .

[119]  Ernest Friedman Hill,et al.  Jess in Action: Java Rule-Based Systems , 2003 .

[120]  Heidrun Schumann,et al.  Visualizing time-oriented data - A systematic view , 2007, Comput. Graph..

[121]  Gary Ewing,et al.  Mismatched Concepts In A Neonatal Intensive Care Unit (NICU): Further Issues For Computer Decision Support? , 2004, Journal of Clinical Monitoring and Computing.

[122]  Chris Mellish,et al.  A Reference Architecture for Natural Language Generation Systems , 2006, Natural Language Engineering.

[123]  Anja Belz,et al.  System Building Cost vs. Output Quality in Data-to-Text Generation , 2009, ENLG.

[124]  Silvia Miksch,et al.  Visualization methods for data analysis and planning in medical applications , 2002, Int. J. Medical Informatics.

[125]  Lawrence M. Fagan,et al.  Combining Physiologic Models and Symbolic Methods to Interpret Time-Varying Patient Data* , 1991, Methods of Information in Medicine.

[126]  van DeemterKees Generating Referring Expressions that Involve Gradable Properties , 2006 .

[127]  P. Sánchez,et al.  VISUALIZATION METHODS FOR TIME-DEPENDENT DATA-AN OVERVIEW , 2003 .

[128]  Chris Mellish,et al.  Choosing the content of textual summaries of large time-series data sets , 2006, Natural Language Engineering.

[129]  Alain Polguère,et al.  Generation of Extended Bilingual Statistical Reports , 1992, COLING.

[130]  Stuart J. Russell,et al.  Probabilistic detection of short events, with application to critical care monitoring , 2008, NIPS.

[131]  R. Kibble Cb or not Cb? Centering theory applied to NLG , 1999 .

[132]  Anja Belz,et al.  Comparing Automatic and Human Evaluation of NLG Systems , 2006, EACL.

[133]  Ehud Reiter,et al.  Generating basic skills reports for low-skilled readers* , 2008, Natural Language Engineering.

[134]  Caroline Ramsey,et al.  Narrative , 2005, Studying The Lord of the Rings.

[135]  Leo Wanner,et al.  Automatic Production of Multilingual Environmental Information , 2007, EnviroInfo.

[136]  E. Tufte,et al.  Graphical summary of patient status , 1994, The Lancet.

[137]  W. Muller,et al.  Visualization methods for time-dependent data - an overview , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[138]  Stephan Busemann ENLG-07 : 11th European Workshop on Natural Language Generation - Proceedings - ; Schloss Dagstuhl, Germany, 17-20 June 2007 , 2007 .