Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue

Factual correctness is often the limiting factor in practical applications of natural language generation in high-stakes domains such as healthcare. An essential requirement for maintaining factuality is the ability to deal with rare tokens. This paper focuses on rare tokens that appear in both the source and the reference sequences, and which, when missed during generation, decrease the factual correctness of the output text. For high-stake domains that are also knowledge-rich, we show how to use knowledge to (a) identify which rare tokens that appear in both source and reference are important and (b) uplift their conditional probability. We introduce the “utilization rate” that encodes knowledge and serves as a regularizer by maximizing the marginal probability of selected tokens. We present a study in a knowledge-rich domain of healthcare, where we tackle the problem of generating after-visit care instructions based on patient-doctor dialogues. We verify that, in our dataset, specific medical concepts with high utilization rates are underestimated by conventionally trained sequence-to-sequence models. We observe that correcting this with our approach to knowledge injection reduces the uncertainty of the model as well as improves factuality and coherence without negatively impacting fluency.

[1]  Yang Gao,et al.  PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization , 2022, COLING.

[2]  Barbara Plank,et al.  Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning , 2022, J. Artif. Intell. Res..

[3]  Hyungjong Noh,et al.  May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation , 2021, ArXiv.

[4]  Kyunghyun Cho,et al.  Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling , 2021, AACL.

[5]  Namit Katariya,et al.  MEDCOD: A Medically-Accurate, Emotive, Diverse, and Controllable Dialog System , 2021, ML4H@NeurIPS.

[6]  Shuohang Wang,et al.  Dict-BERT: Enhancing Language Model Pre-training with Dictionary , 2021, Findings.

[7]  Zhengyuan Liu,et al.  Controllable Neural Dialogue Summarization with Personal Named Entity Planning , 2021, EMNLP.

[8]  Elena Sofia Ruzzetti,et al.  Lacking the Embedding of a Word? Look it up into a Traditional Dictionary , 2021, FINDINGS.

[9]  Pascual Martínez-Gómez,et al.  End-to-End Conversational Search for Online Shopping with Utterance Transfer , 2021, EMNLP.

[10]  An Nguyen,et al.  Language Model Evaluation in Open-ended Text Generation , 2021, ArXiv.

[11]  N. Heffernan,et al.  MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education , 2021, ArXiv.

[12]  Namit Katariya,et al.  Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization , 2021, NLPMC.

[13]  Serena Yeung,et al.  Medical symptom recognition from patient text: An active learning approach for long-tailed multilabel distributions , 2020, ArXiv.

[14]  James J. Zhu,et al.  SmartTriage: A system for personalized patient data capture, documentation generation, and decision support , 2020, ML4H@NeurIPS.

[15]  Xavier Amatriain,et al.  Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures. , 2020, FINDINGS.

[16]  Fuzhen Zhuang,et al.  E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce , 2020, 2009.02835.

[17]  Song Xu,et al.  Self-Attention Guided Copy Mechanism for Abstractive Summarization , 2020, ACL.

[18]  Jeffrey P. Bigham,et al.  Generating SOAP Notes from Doctor-Patient Conversations , 2020, ArXiv.

[19]  Chenliang Li,et al.  Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders , 2019, ACL.

[20]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[21]  David Sontag,et al.  Open Set Medical Diagnosis , 2019, ArXiv.

[22]  Lav R. Varshney,et al.  CTRL: A Conditional Transformer Language Model for Controllable Generation , 2019, ArXiv.

[23]  Jason Weston,et al.  Neural Text Generation with Unlikelihood Training , 2019, ICLR.

[24]  Yejin Choi,et al.  The Curious Case of Neural Text Degeneration , 2019, ICLR.

[25]  Kilian Q. Weinberger,et al.  BERTScore: Evaluating Text Generation with BERT , 2019, ICLR.

[26]  Joe G. Saliby.  Survey on Natural Language Generation , 2019, International Journal of Trend in Scientific Research and Development.

[27]  Myle Ott,et al.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling , 2019, NAACL.

[28]  Alexander M. Rush,et al.  Bottom-Up Abstractive Summarization , 2018, EMNLP.

[29]  Myle Ott,et al.  Scaling Neural Machine Translation , 2018, WMT.

[30]  Matt Post,et al.  Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation , 2018, NAACL.

[31]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[32]  Qun Liu,et al.  Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search , 2017, ACL.

[33]  Jeff Johnson,et al.  Billion-Scale Similarity Search with GPUs , 2017, IEEE Transactions on Big Data.

[34]  Matthijs Douze,et al.  FastText.zip: Compressing text classification models , 2016, ArXiv.

[35]  Alexandra Birch,et al.  Neural Machine Translation of Rare Words with Subword Units , 2015, ACL.

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Quoc V. Le,et al.  Addressing the Rare Word Problem in Neural Machine Translation , 2014, ACL.

[38]  N. McKeown,et al.  A New Direction for Machine Learning in Criminal Law , 2021 .

[39]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[40]  Olivier Bodenreider,et al.  The Unified Medical Language System (UMLS): integrating biomedical terminology , 2004, Nucleic Acids Res..