Attention-Based Abnormal-Aware Fusion Network for Radiology Report Generation

Radiology report writing is error-prone, time-consuming and tedious for radiologists. Medical reports are usually dominated by a large number of normal findings, and the abnormal findings are few but more important. Current report generation methods often fail to depict these prominent abnormal findings. In this paper, we propose a model named Attention-based Abnormal-Aware Fusion Network (A3FN). We break down sentence generation into abnormal and normal sentence generation through a high level gate module. We also adopt a topic guide attention mechanism for better capturing visual details and develop a context-aware topic vector for model cross-sentence topic coherence. Experiments on real radiology image datasets demonstrate the effectiveness of our proposed method.