Improving Radiology Summarization with Radiograph and Anatomy Prompts

The impression is crucial for the referring physicians to grasp key information since it is concluded from the findings and reasoning of radiologists. To alleviate the workload of radiologists and reduce repetitive human labor in impression writing, many researchers have focused on automatic impression generation. However, recent works on this task mainly summarize the corresponding findings and pay less attention to the radiology images. In clinical, radiographs can provide more detailed valuable observations to enhance radiologists’ impression writing, especially for complicated cases. Besides, each sentence in findings usually focuses on single anatomy, so they only need to be matched to corresponding anatomical regions instead of the whole image, which is beneficial for textual and visual features alignment. Therefore, we propose a novel anatomy-enhanced multimodal model to promote impression generation. In detail, we first construct a set of rules to extract anatomies and put these prompts into each sentence to highlight anatomy characteristics. Then, two separate encoders are applied to extract features from the radiograph and findings. After-ward, we utilize a contrastive learning module to align these two representations at the overall level and use a co-attention to fuse them at the sentence level with the help of anatomy-enhanced sentence representation. Finally, the decoder takes the fused information as the input to generate impressions. The experimental results on two benchmark datasets confirm the effectiveness of the proposed method, which achieves state-of-the-art results.

[1]  Shen Ge,et al.  Competence-based Multimodal Curriculum Learning for Medical Report Generation , 2022, ACL.

[2]  Yan Song,et al.  Cross-modal Memory Networks for Radiology Report Generation , 2022, ACL.

[3]  Tsung-Hui Chang,et al.  Graph Enhanced Contrastive Learning for Radiology Findings Summarization , 2022, ACL.

[4]  Shen Ge,et al.  AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation , 2022, MICCAI.

[5]  Sanjeev Kumar Karn,et al.  Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarization , 2022, ACL.

[6]  Tsung-Hui Chang,et al.  Word Graph Guided Summarization for Radiology Findings , 2021, FINDINGS.

[7]  H. Fu,et al.  Supplementary Document: Visual-Textual Attentive Semantic Consistency for Medical Report Generation , 2021 .

[8]  Achleshwar Luthra,et al.  MedSkip: Medical Report Generation Using Skip Connections and Integrated Attention , 2021, 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).

[9]  Hyunsouk Cho,et al.  Self-Supervised Multimodal Opinion Summarization , 2021, ACL.

[10]  Yash Kumar Atri,et al.  See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization , 2021, Knowl. Based Syst..

[11]  Qi Wu,et al.  Towards Accurate Text-based Image Captioning with Content Diversity Exploration , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Danqi Chen,et al.  SimCSE: Simple Contrastive Learning of Sentence Embeddings , 2021, EMNLP.

[13]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[14]  Jiajun Zhang,et al.  Multimodal Sentence Summarization via Multimodal Selective Encoding , 2020, COLING.

[15]  Tsung-Hui Chang,et al.  Generating Radiology Reports via Memory-driven Transformer , 2020, EMNLP.

[16]  Christopher D. Manning,et al.  Contrastive Learning of Medical Visual Representations from Paired Images and Text , 2020, MLHC.

[17]  Nazli Goharian,et al.  Attend to Medical Ontologies: Content Selection for Clinical Abstractive Summarization , 2020, ACL.

[18]  William Boag,et al.  Baselines for Chest X-Ray Report Generation , 2020, ML4H@NeurIPS.

[19]  Andrew Y. Ng,et al.  CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT , 2020, EMNLP.

[20]  Yu Zhou,et al.  Multimodal Summarization with Guidance of Multimodal Reference , 2020, AAAI.

[21]  Daguang Xu,et al.  When Radiology Report Generation Meets Knowledge Graph , 2020, AAAI.

[22]  Christopher D. Manning,et al.  Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports , 2019, ACL.

[23]  Yue Zhang,et al.  Contrastive Attention Mechanism for Abstractive Sentence Summarization , 2019, EMNLP.

[24]  Mirella Lapata,et al.  Text Summarization with Pretrained Encoders , 2019, EMNLP.

[25]  Eric P. Xing,et al.  Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports , 2019, ACL.

[26]  Xiaodong He,et al.  Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations , 2019, NeurIPS.

[27]  Nazli Goharian,et al.  Ontology-Aware Clinical Abstractive Summarization , 2019, SIGIR.

[28]  Franck Dernoncourt,et al.  Scoring Sentence Singletons and Pairs for Abstractive Summarization , 2019, ACL.

[29]  Peter Szolovits,et al.  Clinically Accurate Chest X-Ray Report Generation , 2019, MLHC.

[30]  Jaewoo Kang,et al.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining , 2019, Bioinform..

[31]  Christopher D. Manning,et al.  Learning to Summarize Radiology Findings , 2018, Louhi@EMNLP.

[32]  Haoran Li,et al.  Multi-modal Sentence Summarization with Modality Attention and Image Filtering , 2018, IJCAI.

[33]  Yen-Chun Chen,et al.  Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting , 2018, ACL.

[34]  Alexander G. Schwing,et al.  Convolutional Image Captioning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Haoran Li,et al.  Multi-modal Summarization for Asynchronous Collection of Text, Image, Audio and Video , 2017, EMNLP.

[36]  Christopher D. Manning,et al.  Get To The Point: Summarization with Pointer-Generator Networks , 2017, ACL.

[37]  Jiebo Luo,et al.  Image Captioning with Semantic Attention , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Clement J. McDonald,et al.  Preparing a collection of radiology examinations for distribution and retrieval , 2015, J. Am. Medical Informatics Assoc..

[40]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[41]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[42]  Tsung-Hui Chang,et al.  Exploring Word Segmentation and Medical Concept Recognition for Chinese Medical Texts , 2021, BIONLP.

[43]  Jean-Benoit Delbrouck,et al.  QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization , 2021, BIONLP.

[44]  Yu Zhou,et al.  MSMO: Multimodal Summarization with Multimodal Output , 2018, EMNLP.