Boosting Radiology Report Generation by Infusing Comparison Prior

Current transformer-based models achieved great success in generating radiology reports from chest X-ray images. Nonetheless, one of the major issues is the model's lack of prior knowledge, which frequently leads to false references to non-existent prior exams in synthetic reports. This is mainly due to the knowledge gap between radiologists and the generation models: radiologists are aware of the prior information of patients to write a medical report, while models only receive X-ray images at a specific time. To address this issue, we propose a novel approach that employs a labeler to extract comparison prior information from radiology reports in the IU X-ray and MIMIC-CXR datasets. This comparison prior is then incorporated into state-of-the-art transformer-based models, allowing them to generate more realistic and comprehensive reports. We test our method on the IU X-ray and MIMIC-CXR datasets and find that it outperforms previous state-of-the-art models in terms of both automatic and human evaluation metrics. In addition, unlike previous models, our model generates reports that do not contain false references to non-existent prior exams. Our approach provides a promising direction for bridging the gap between radiologists and generation models in medical report generation.

[1]  P. Rajpurkar,et al.  Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors , 2022, ML4H@NeurIPS.

[2]  M. Montes-y-Gómez,et al.  Medical Report Generation through Radiology Images: An Overview. , 2022, IEEE Latin America Transactions.

[3]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[4]  Michael Krauthammer,et al.  Progressive Transformer-Based Generation of Radiology Reports , 2021, EMNLP.

[5]  Yuhao Zhang,et al.  Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation , 2020, NAACL.

[6]  Andrew Y. Ng,et al.  Retrieval-Based Chest X-Ray Report Generation Using a Pre-trained Contrastive Language-Image Model , 2021, ML4H@NeurIPS.

[7]  Bobak Mortazavi,et al.  Learning to Generate Clinically Coherent Chest X-Ray Reports , 2020, FINDINGS.

[8]  Tsung-Hui Chang,et al.  Generating Radiology Reports via Memory-driven Transformer , 2020, EMNLP.

[9]  Marcella Cornia,et al.  Meshed-Memory Transformer for Image Captioning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Qinghua Zheng,et al.  Automatic Generation of Medical Imaging Diagnostic Report with Hierarchical Recurrent Neural Network , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[11]  Jiebo Luo,et al.  Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment , 2019, MICCAI.

[12]  Peter Szolovits,et al.  Clinically Accurate Chest X-Ray Report Generation , 2019, MLHC.

[13]  Eric P. Xing,et al.  Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generation , 2019, AAAI.

[14]  Yifan Yu,et al.  CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison , 2019, AAAI.

[15]  Shuang Bai,et al.  A survey on automatic image caption generation , 2018, Neurocomputing.

[16]  Tao Xu,et al.  Multimodal Recurrent Model with Attention for Automated Radiology Report Generation , 2018, MICCAI.

[17]  Qingyang Xu,et al.  A survey on deep neural network-based image captioning , 2018, The Visual Computer.

[18]  Ronald M. Summers,et al.  TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Pengtao Xie,et al.  On the Automatic Generation of Medical Imaging Reports , 2017, ACL.

[20]  Lin Yang,et al.  MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Clement J. McDonald,et al.  Preparing a collection of radiology examinations for distribution and retrieval , 2015, J. Am. Medical Informatics Assoc..

[24]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[25]  C. Lawrence Zitnick,et al.  CIDEr: Consensus-based image description evaluation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[27]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[28]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.