Deep Learning Based Decision Support for Medicine - A Case Study on Skin Cancer Diagnosis

Early detection of skin cancers like melanoma is crucial to ensure high chances of survival for patients. Clinical application of Deep Learning (DL)-based Decision Support Systems (DSS) for skin cancer screening has the potential to improve the quality of patient care. The majority of work in the medical AI community focuses on a diagnosis setting that is mainly relevant for autonomous operation. Practical decision support should, however, go beyond plain diagnosis and provide explanations. This paper provides an overview of works towards explainable, DL-based decision support in medical applications with the example of skin cancer diagnosis from clinical, dermoscopic and histopathologic images. Analysis reveals that comparably little attention is payed to the explanation of histopathologic skin images and that current work is dominated by visual relevance maps as well as dermoscopic feature identification. We conclude that future work should focus on meeting the stakeholder’s cognitive concepts, providing exhaustive explanations that combine global and local approaches and leverage diverse modalities. Moreover, the possibility to intervene and guide models in case of misbehaviour is identified as a major step towards successful deployment of AI as DL-based DSS and beyond.

[1]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Yixin Chen,et al.  SHOW , 2018, Silent Cinema.

[3]  Wojciech Samek,et al.  Methods for interpreting and understanding deep neural networks , 2017, Digit. Signal Process..

[4]  Andreas Holzinger,et al.  The European Legal Framework for Medical AI , 2020, CD-MAKE.

[5]  Fabio A. González,et al.  Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks , 2014, Medical Imaging.

[6]  Sébastien Ourselin,et al.  CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation , 2021, IEEE Transactions on Medical Imaging.

[7]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[8]  Been Kim,et al.  Sanity Checks for Saliency Maps , 2018, NeurIPS.

[9]  Chris Russell,et al.  Explaining Explanations in AI , 2018, FAT.

[10]  Yong Xia,et al.  Attention Residual Learning for Skin Lesion Classification , 2019, IEEE Transactions on Medical Imaging.

[11]  Francisco Herrera,et al.  Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI , 2020, Inf. Fusion.

[12]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[13]  Tim Verbelen,et al.  Visualizing Convolutional Neural Networks to Improve Decision Support for Skin Lesion Classification , 2018, MLCN/DLF/iMIMIC@MICCAI.

[14]  Hitoshi Iyatomi,et al.  Towards Explainable Melanoma Diagnosis: Prediction of Clinical Indicators Using Semi-supervised and Multi-task Learning , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[15]  Muhammad Naseer Bajwa,et al.  Explaining AI-based Decision Support Systems using Concept Localization Maps , 2020, ICONIP.

[16]  Alexander Binder,et al.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , 2015, PloS one.

[17]  Jorge S. Marques,et al.  Explainable skin lesion diagnosis using taxonomies , 2021, Pattern Recognit..

[18]  Joann G Elmore,et al.  Pathologists ' diagnosis of invasive melanoma and melanocytic proliferations : observer accuracy and reproducibility study , 2019 .

[19]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[20]  F. Ghorat,et al.  GLOBAL INCIDENCE AND MORTALITY OF SKIN CANCER BY HISTOLOGICAL SUBTYPE AND ITS RELATIONSHIP WITH THE HUMAN DEVELOPMENT INDEX (HDI); , 2019 .

[21]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[23]  Daniel Sonntag,et al.  The Skincare project, an interactive deep learning system for differential diagnosis of malignant skin lesions. Technical Report , 2020, arXiv.org.

[24]  Ghassan Hamarneh,et al.  Melanoma Recognition via Visual Attention , 2019, IPMI.

[25]  Rongrong Ji,et al.  Architecture Disentanglement for Deep Neural Networks , 2020, ArXiv.

[26]  Lin Yang,et al.  MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Klaus-Robert Müller,et al.  Resolving challenges in deep learning-based analyses of histopathological images using explanation methods , 2019, Scientific Reports.

[28]  Rahil Garnavi,et al.  Skin Disease Recognition Using Deep Saliency Features and Multimodal Learning of Dermoscopy and Clinical Images , 2017, MICCAI.

[29]  Jun Akatsuka,et al.  Automated acquisition of explainable knowledge from unannotated histopathology images , 2019, Nature Communications.

[30]  Xiaoxiao Li,et al.  What evidence does deep learning model use to classify Skin Lesions? , 2018, ArXiv.

[31]  Fabio A. González,et al.  A Deep Learning Architecture for Image Representation, Visual Interpretability and Automated Basal-Cell Carcinoma Cancer Detection , 2013, MICCAI.

[32]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Fei Wang,et al.  Towards Interpretable Skin Lesion Classification with Deep Learning Models , 2019, AMIA.

[35]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[36]  Chandan Singh,et al.  Interpretations are useful: penalizing explanations to align neural networks with prior knowledge , 2019, ICML.

[37]  John R. Smith,et al.  Collaborative Human-AI (CHAI): Evidence-Based Interpretable Melanoma Classification in Dermoscopic Images , 2018, MLCN/DLF/iMIMIC@MICCAI.

[38]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[39]  Orcun Goksel,et al.  A Structure-Aware Convolutional Neural Network for Skin Lesion Classification , 2018, OR 2.0/CARE/CLIP/ISIC@MICCAI.

[40]  Sally Shrapnel,et al.  Deep neural network or dermatologist? , 2019, iMIMIC/ML-CDS@MICCAI.

[41]  Eduardo Valle,et al.  (De) Constructing Bias on Skin Lesion Datasets , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[42]  Muhammad Awais,et al.  Medical image retrieval using deep convolutional neural network , 2017, Neurocomputing.

[43]  Martin Wattenberg,et al.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) , 2017, ICML.

[44]  LinLin Shen,et al.  Skin Lesion Classification using Class Activation Map , 2017, ArXiv.

[45]  Theo Gevers,et al.  Integrating Clinically-Relevant Features into Skin Lesion Classification , 2019, BNAIC/BENELEARN.

[46]  Pitoyo Hartono,et al.  Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images , 2020, J. Biomed. Informatics.

[47]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[48]  W. Stolz,et al.  The ABCD rule of dermatoscopy. High prospective value in the diagnosis of doubtful melanocytic skin lesions. , 1994, Journal of the American Academy of Dermatology.

[49]  Zhiguo Jiang,et al.  Classification for Dermoscopy Images Using Convolutional Neural Networks Based on Region Average Pooling , 2018, IEEE Access.

[50]  Kai Lu,et al.  Interpretable Classification from Skin Cancer Histology Slides Using Deep Learning: A Retrospective Multicenter Study , 2019, ArXiv.

[51]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[52]  C. Rudin,et al.  Concept whitening for interpretable image recognition , 2020, Nature Machine Intelligence.

[53]  D. Lansing Taylor,et al.  HistoMapr™: An Explainable AI (xAI) Platform for Computational Pathology Solutions , 2020, AI and ML for Digital Pathology.

[54]  Ji Wan,et al.  Deep Learning for Content-Based Image Retrieval: A Comprehensive Study , 2014, ACM Multimedia.

[55]  Cynthia Rudin,et al.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead , 2018, Nature Machine Intelligence.

[56]  Toshihiko Yamasaki,et al.  Interpreting Fine-Grained Dermatological Classification by Deep Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[57]  Muhammad Naseer Bajwa,et al.  On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[58]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[59]  Ghassan Hamarneh,et al.  Seven-Point Checklist and Skin Lesion Classification Using Multitask Multimodal Neural Nets , 2019, IEEE Journal of Biomedical and Health Informatics.

[60]  Ghassan Hamarneh,et al.  Illumination-based Transformations Improve Skin Lesion Segmentation in Dermoscopic Images , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[61]  Bjorn Ommer,et al.  A Disentangling Invertible Interpretation Network for Explaining Latent Representations , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Subhransu Maji,et al.  Bilinear CNN Models for Fine-Grained Visual Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[63]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[64]  Amit Dhurandhar,et al.  One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques , 2019, ArXiv.

[65]  Daniel L. Rubin,et al.  Regulatory Frameworks for Development and Evaluation of Artificial Intelligence–Based Diagnostic Imaging Algorithms: Summary and Recommendations , 2020, Journal of the American College of Radiology.

[66]  Achim Hekler,et al.  Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. , 2019, European journal of cancer.

[67]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[68]  M. Abràmoff,et al.  Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices , 2018, npj Digital Medicine.

[69]  Wei-Shi Zheng,et al.  Biomarker Localization by Combining CNN Classifier and Generative Adversarial Network , 2019, MICCAI.

[70]  G. Argenziano,et al.  Epiluminescence microscopy for the diagnosis of doubtful melanocytic skin lesions. Comparison of the ABCD rule of dermatoscopy and a new 7-point checklist based on pattern analysis. , 1998, Archives of dermatology.

[71]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[72]  Xiaoping Zhou,et al.  Development and validation of an interpretable deep learning framework for Alzheimer’s disease classification , 2020, Brain : a journal of neurology.

[73]  Ghassan Hamarneh,et al.  Fully Convolutional Neural Networks to Detect Clinical Dermoscopic Features , 2017, IEEE Journal of Biomedical and Health Informatics.

[74]  Tao Mei,et al.  Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Cuntai Guan,et al.  Interpreting mechanisms of prediction for skin cancer diagnosis using multi-task learning , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).