Explainable artificial intelligence (XAI) in radiology and nuclear medicine: a literature review

Rational Deep learning (DL) has demonstrated a remarkable performance in diagnostic imaging for various diseases and modalities and therefore has a high potential to be used as a clinical tool. However, current practice shows low deployment of these algorithms in clinical practice, because DL algorithms lack transparency and trust due to their underlying black-box mechanism. For successful employment, explainable artificial intelligence (XAI) could be introduced to close the gap between the medical professionals and the DL algorithms. In this literature review, XAI methods available for magnetic resonance (MR), computed tomography (CT), and positron emission tomography (PET) imaging are discussed and future suggestions are made. Methods PubMed, Embase.com and Clarivate Analytics/Web of Science Core Collection were screened. Articles were considered eligible for inclusion if XAI was used (and well described) to describe the behavior of a DL model used in MR, CT and PET imaging. Results A total of 75 articles were included of which 54 and 17 articles described post and ad hoc XAI methods, respectively, and 4 articles described both XAI methods. Major variations in performance is seen between the methods. Overall, post hoc XAI lacks the ability to provide class-discriminative and target-specific explanation. Ad hoc XAI seems to tackle this because of its intrinsic ability to explain. However, quality control of the XAI methods is rarely applied and therefore systematic comparison between the methods is difficult. Conclusion There is currently no clear consensus on how XAI should be deployed in order to close the gap between medical professionals and DL algorithms for clinical implementation. We advocate for systematic technical and clinical quality assessment of XAI methods. Also, to ensure end-to-end unbiased and safe integration of XAI in clinical workflow, (anatomical) data minimization and quality control methods should be included.

[1]  D. Dey,et al.  Deep Learning for Explainable Estimation of Mortality Risk From Myocardial Positron Emission Tomography Images , 2022, Circulation. Cardiovascular imaging.

[2]  B. Metin,et al.  A Class Activation Map-Based Interpretable Transfer Learning Model for Automated Detection of ADHD from fMRI Data , 2022, Clinical EEG and neuroscience.

[3]  À. Rovira,et al.  Detection of lesions in the optic nerve with magnetic resonance imaging using a 3D convolutional neural network , 2022, NeuroImage: Clinical.

[4]  Derya Yakar,et al.  AI-assisted biparametric MRI surveillance of prostate cancer: feasibility study , 2022, European Radiology.

[5]  Yicong Tian,et al.  Inverted papilloma and nasal polyp classification using a deep convolutional network integrated with an attention mechanism , 2022, Comput. Biol. Medicine.

[6]  Bingbing Ni,et al.  GMILT: A Novel Transformer Network That Can Noninvasively Predict EGFR Mutation Status. , 2022, IEEE transactions on neural networks and learning systems.

[7]  Md. Golam Rabiul Alam,et al.  Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from CT-radiography , 2022, Scientific Reports.

[8]  Xiang Li,et al.  Explainable multi-instance and multi-task learning for COVID-19 diagnosis and lesion segmentation in CT images , 2022, Knowledge-Based Systems.

[9]  C. Caltagirone,et al.  A Reproducible Deep-Learning-Based Computer-Aided Diagnosis Tool for Frontotemporal Dementia Using MONAI and Clinica Frameworks , 2022, Life.

[10]  Sohail Jabbar,et al.  Explainable artificial intelligence approach in combating real-time surveillance of COVID19 pandemic from CT scan and X-ray images using ensemble model , 2022, The Journal of Supercomputing.

[11]  S. Majumdar,et al.  Automatic detection and voxel‐wise mapping of lumbar spine Modic changes with deep learning , 2022, JOR spine.

[12]  David W. Sobel,et al.  COVLIAS 2.0-cXAI: Cloud-Based Explainable Deep Learning System for COVID-19 Lesion Localization in Computed Tomography Scans , 2022, Diagnostics.

[13]  M. E. Karar,et al.  Explainability of deep neural networks for MRI analysis of brain tumors , 2022, International Journal of Computer Assisted Radiology and Surgery.

[14]  Xiaojie Huang,et al.  SSA-Net: Spatial self-attention network for COVID-19 pneumonia infection segmentation with semi-supervised few-shot learning , 2022, Medical Image Analysis.

[15]  Ronald X. Xu,et al.  A Single Model Deep Learning Approach for Alzheimer’s Disease Diagnosis , 2022, Neuroscience.

[16]  Gurmail Singh.  Think positive: An interpretable neural network for image recognition , 2022, Neural Networks.

[17]  Saurav Mallik,et al.  Explanation-Driven Deep Learning Model for Prediction of Brain Tumour Status Using MRI Image Data , 2022, Frontiers in Genetics.

[18]  N. Dosenbach,et al.  BrainQCNet: a Deep Learning attention-based model for multi-scale detection of artifacts in brain structural MRI scans , 2022, bioRxiv.

[19]  I. Burger,et al.  Artificial Intelligence Applications on Restaging [18F]FDG PET/CT in Metastatic Colorectal Cancer: A Preliminary Report of Morpho-Functional Radiomics Classification for Prediction of Disease Outcome , 2022, Applied Sciences.

[20]  Kyle J. Lafata,et al.  A Neural Ordinary Differential Equation Model for Visualizing Deep Neural Network Behaviors in Multi-Parametric MRI based Glioma Segmentation , 2022, Medical physics.

[21]  Li-Zhong Wu,et al.  MultiR-Net: A Novel Joint Learning Network for COVID-19 segmentation and classification , 2022, Computers in Biology and Medicine.

[22]  V. Calhoun,et al.  SSPNet: An interpretable 3D-CNN for classification of schizophrenia using phase maps of resting-state complex-valued fMRI data , 2022, Medical Image Anal..

[23]  Wei-min Li,et al.  Predicting EGFR and PD-L1 Status in NSCLC Patients Using Multitask AI System Based on CT Images , 2022, Frontiers in Immunology.

[24]  Ayisha Al Busaidi,et al.  Deep learning models for triaging hospital head MRI examinations , 2022, Medical Image Anal..

[25]  Brady J. Williamson,et al.  Automated grading of enlarged perivascular spaces in clinical imaging data of an acute stroke cohort using an interpretable, 3D deep learning framework , 2022, Scientific Reports.

[26]  T. Brox,et al.  Explainable AI for CNN-based prostate tumor segmentation in multi-parametric MRI correlated to whole mount histopathology , 2022, Radiation oncology.

[27]  Ronald M. Summers,et al.  Trustworthy Artificial Intelligence in Medical Imaging. , 2022, PET clinics.

[28]  Shuihua Wang,et al.  CGENet: A Deep Graph Model for COVID-19 Detection Based on Chest CT , 2021, Biology.

[29]  Jianyang Xie,et al.  Automatic Sequence-Based Network for Lung Diseases Detection in Chest CT , 2021, Frontiers in Oncology.

[30]  C. Klaver,et al.  Trustworthy AI: Closing the gap between development and integration of AI systems in ophthalmic practice , 2021, Progress in Retinal and Eye Research.

[31]  S. Vandenberghe,et al.  Artificial intelligence with deep learning in nuclear medicine and radiology , 2021, EJNMMI Physics.

[32]  L. Carin,et al.  Explainable multiple abnormality classification of chest CT volumes with AxialNet and HiResCAM , 2021, Artif. Intell. Medicine.

[33]  Jordan D. Fuhrman,et al.  A review of explainable and interpretable AI with applications in COVID‐19 imaging , 2021, Medical physics.

[34]  Sattam Al Otaibi,et al.  An Analysis of Image Features Extracted by CNNs to Design Classification Models for COVID-19 and Non-COVID-19 , 2021, Journal of Signal Processing Systems.

[35]  Dominik F. Bauer,et al.  Automated Screening for Abdominal Aortic Aneurysm in CT Scans under Clinical Conditions Using Deep Learning , 2021, Diagnostics.

[36]  J. Geitung,et al.  Explainable Artificial Intelligence for Human-Machine Interaction in Brain Tumor Localization , 2021, Journal of personalized medicine.

[37]  S. Satapathy,et al.  Doctor's Dilemma: Evaluating an Explainable Subtractive Spatial Lightweight Convolutional Neural Network for Brain Tumor Diagnosis , 2021, ACM Trans. Multim. Comput. Commun. Appl..

[38]  Hongbing Lu,et al.  IMAL-Net: Interpretable Multi-task Attention Learning Network for Invasive Lung Adenocarcinoma Screening in CT Images. , 2021, Medical physics.

[39]  O. Speck,et al.  TorchEsegeta: Framework for Interpretability and Explainability of Image-based Deep Learning Models , 2021, Applied Sciences.

[40]  R. Cuocolo,et al.  Predictive Medicine for Salivary Gland Tumours Identification Through Deep Learning , 2021, IEEE Journal of Biomedical and Health Informatics.

[41]  M. Chikh,et al.  Deep transfer learning based classification model for covid-19 using chest CT-scans , 2021, Pattern Recognition Letters.

[42]  K. Yow,et al.  Object or Background: An Interpretable Deep Learning Model for COVID-19 Detection from CT-Scan Images , 2021, Diagnostics.

[43]  Z. Siddique,et al.  Detection of COVID-19 Patients from CT Scan and Chest X-ray Data Using Modified MobileNetV2 and LIME , 2021, Healthcare.

[44]  A. Kaufman,et al.  COVID-view: Diagnosis of COVID-19 using Chest CT , 2021, IEEE Transactions on Visualization and Computer Graphics.

[45]  Iam Palatnik de Sousa,et al.  Explainable Artificial Intelligence for Bias Detection in COVID CT-Scan Classifiers , 2021, Sensors.

[46]  Jayanthi Sivaswamy,et al.  Lung nodule malignancy classification with weakly supervised explanation generation , 2021, Journal of medical imaging.

[47]  L. Peng,et al.  Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence , 2021, BMJ Open.

[48]  J. Zhou,et al.  Structural and diffusion MRI based schizophrenia classification using 2D pretrained and 3D naive Convolutional Neural Networks , 2021, Schizophrenia Research.

[49]  Kwang Hyeon Kim,et al.  Cerebral hemorrhage detection and localization with medical imaging for cerebrovascular disease diagnosis and treatment using explainable deep learning , 2021, Journal of the Korean Physical Society.

[50]  Mehmed Kantardzic,et al.  A Review of Explainable Deep Learning Cancer Detection Models in Medical Imaging , 2021, Applied Sciences.

[51]  P. Dragotti,et al.  Joint Learning of 3D Lesion Segmentation and Classification for Explainable COVID-19 Diagnosis , 2021, IEEE Transactions on Medical Imaging.

[52]  May D. Wang,et al.  COVID-19 Automatic Diagnosis With Radiographic Imaging: Explainable Attention Transfer Deep Neural Networks , 2021, IEEE Journal of Biomedical and Health Informatics.

[53]  Basit Raza,et al.  Visual interpretability in 3D brain tumor segmentation network , 2021, Comput. Biol. Medicine.

[54]  Yu-Dong Zhang,et al.  Explainable diagnosis of secondary pulmonary tuberculosis by graph rank-based average pooling neural network , 2021 .

[55]  R. Mak,et al.  Deep learning classification of lung cancer histology using CT images , 2021, Scientific Reports.

[56]  F. Zanca,et al.  The EU medical device regulation: Implications for artificial intelligence-based medical device software in medical physics. , 2021, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[57]  Simone Palazzo,et al.  An explainable AI system for automated COVID-19 assessment and lesion categorization from CT-scans , 2021, Artificial Intelligence in Medicine.

[58]  Dominique Duncan,et al.  Efficient and visualizable convolutional neural networks for COVID-19 classification using Chest CT , 2020, Expert Systems with Applications.

[59]  J. Reichenbach,et al.  Investigation of Deep-Learning-Driven Identification of Multiple Sclerosis Patients Based on Susceptibility-Weighted Images Using Relevance Analysis , 2020, Frontiers in Neuroscience.

[60]  Thomas Martinetz,et al.  Explainable COVID-19 Detection Using Chest CT Scans and Deep Learning , 2020, Sensors.

[61]  Weilun Wang,et al.  Automatic prognosis of lung cancer using heterogeneous deep learning models for nodule detection and eliciting its morphological features , 2020, Appl. Intell..

[62]  Markus H. A. Janse,et al.  Volumetric breast density estimation on MRI using explainable deep learning regression , 2020, Scientific Reports.

[63]  Huazhu Fu,et al.  M$^3$Lung-Sys: A Deep Learning System for Multi-Class Lung Pneumonia Screening From CT Imaging , 2020, IEEE Journal of Biomedical and Health Informatics.

[64]  Tom Vercauteren,et al.  CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation , 2020, IEEE Transactions on Medical Imaging.

[65]  W. M. van der Flier,et al.  Classification of negative and positive 18F-florbetapir brain PET studies in subjective cognitive decline patients using a convolutional neural network , 2020, European Journal of Nuclear Medicine and Molecular Imaging.

[66]  Konstantinos N. Plataniotis,et al.  MIXCAPS: A Capsule Network-based Mixture of Experts for Lung Nodule Malignancy Prediction , 2020, Pattern Recognit..

[67]  Daoqiang Zhang,et al.  An Explainable 3D Residual Self-Attention Deep Neural Network for Joint Atrophy Localization and Alzheimer’s Disease Diagnosis Using Structural MRI , 2020, IEEE Journal of Biomedical and Health Informatics.

[68]  Kishor Datta Gupta,et al.  Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset , 2020, ArXiv.

[69]  Chunshui Yu,et al.  Generalizable, Reproducible, and Neuroscientifically Interpretable Imaging Biomarkers for Alzheimer's Disease , 2020, Advanced science.

[70]  A. Muacevic,et al.  Implementation of model explainability for a basic brain tumor detection using convolutional neural networks on MRI slices , 2020, Neuroradiology.

[71]  Imon Banerjee,et al.  An Automated Two-step Pipeline for Aggressive Prostate Lesion Detection from Multi-parametric MR Sequence. , 2020, AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science.

[72]  Amitojdeep Singh,et al.  Explainable Deep Learning Models in Medical Image Analysis , 2020, J. Imaging.

[73]  M. Zokaeinikoo,et al.  AIDCOV: An Interpretable Artificial Intelligence Model for Detection of COVID-19 from Chest Radiography Images , 2020, medRxiv.

[74]  Wei Zhang,et al.  Accurate Screening of COVID-19 Using Attention-Based Deep 3D Multiple Instance Learning , 2020, IEEE Transactions on Medical Imaging.

[75]  Mohammad Saniee Abadeh,et al.  Brain MRI analysis using a deep learning based evolutionary approach , 2020, Neural Networks.

[76]  Plamen Angelov,et al.  Towards Explainable Deep Neural Networks (xDNN) , 2019, Neural Networks.

[77]  Philipp Berens,et al.  On the ethics of algorithmic decision-making in healthcare , 2019, Journal of Medical Ethics.

[78]  Jun Akatsuka,et al.  Illuminating Clues of Cancer Buried in Prostate MR Image: Deep Learning and Expert Approaches , 2019, Biomolecules.

[79]  Adebayo Felix Adekoya,et al.  Capsule Networks - A survey , 2019, J. King Saud Univ. Comput. Inf. Sci..

[80]  Amisha,et al.  Overview of artificial intelligence in medicine , 2019, Journal of family medicine and primary care.

[81]  J. Duncan,et al.  Deep learning for liver tumor diagnosis part II: convolutional neural network interpretation using radiologic imaging features , 2019, European Radiology.

[82]  Sally L. Baxter,et al.  The practical implementation of artificial intelligence technologies in medicine , 2019, Nature Medicine.

[83]  Alexander D. Weston,et al.  What Does Deep Learning See? Insights From a Classifier Trained to Predict Contrast Enhancement Phase From CT Images. , 2018, AJR. American journal of roentgenology.

[84]  Kate Saenko,et al.  RISE: Randomized Input Sampling for Explanation of Black-box Models , 2018, BMVC.

[85]  Anirban Sarkar,et al.  Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[86]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[87]  Avanti Shrikumar,et al.  Learning Important Features Through Propagating Activation Differences , 2017, ICML.

[88]  Stuart A. Taylor,et al.  Imaging biomarker roadmap for cancer studies , 2016, Nature Reviews Clinical Oncology.

[89]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[90]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[91]  Thanuja D. Ambegoda,et al.  Interpretable machine learning for brain tumour analysis using MRI and whole slide images , 2022, Softw. Impacts.

[92]  H. Habe,et al.  A Deep Learning System to Diagnose COVID-19 Pneumonia Using Masked Lung CT Images to Avoid AI-generated COVID-19 Diagnoses that Include Data outside the Lungs , 2022, Advanced Biomedical Engineering.

[93]  Dietrich Rebholz-Schuhmann,et al.  DeepKneeExplainer: Explainable Knee Osteoarthritis Diagnosis From Radiographs and Magnetic Resonance Imaging , 2021, IEEE Access.

[94]  Ketan Kotecha,et al.  A novel AI-based System for Detection and Severity Prediction of Dementia using MRI , 2021, IEEE Access.

[95]  Klaus-Robert Müller,et al.  Layer-Wise Relevance Propagation: An Overview , 2019, Explainable AI.

[96]  J. Affeldt,et al.  The feasibility study , 2019, The Information System Consultant’s Handbook.