Survey of XAI in Digital Pathology

Artificial intelligence (AI) has shown great promise for diagnostic imaging assessments. However, the application of AI to support medical diagnostics in clinical routine comes with many challenges. The algorithms should have high prediction accuracy but also be transparent, understandable and reliable. Thus, explainable artificial intelligence (XAI) is highly relevant for this domain. We present a survey on XAI within digital pathology, a medical imaging sub-discipline with particular characteristics and needs. The review includes several contributions. Firstly, we give a thorough overview of current XAI techniques of potential relevance for deep learning methods in pathology imaging, and categorise them from three different aspects. In doing so, we incorporate uncertainty estimation methods as an integral part of the XAI landscape. We also connect the technical methods to the specific prerequisites in digital pathology and present findings to guide future research efforts. The survey is intended for both technical researchers and medical professionals, one of the objectives being to establish a common ground for cross-disciplinary discussions.

[1]  Atul Mishra,et al.  A Scalable, Lexicon Based Technique for Sentiment Analysis , 2014, FOCS 2014.

[2]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Pascal Vincent,et al.  Visualizing Higher-Layer Features of a Deep Network , 2009 .

[4]  Nilanjan Ray,et al.  Cell Counting by Regression Using Convolutional Neural Network , 2016, ECCV Workshops.

[5]  T. Hermanns,et al.  Automated Gleason grading of prostate cancer tissue microarrays via deep learning , 2018, Scientific Reports.

[6]  Geert J. S. Litjens,et al.  Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology , 2019, Medical Image Anal..

[7]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[8]  Scott Lundberg,et al.  A Unified Approach to Interpreting Model Predictions , 2017, NIPS.

[9]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[10]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[11]  Chris Russell,et al.  Explaining Explanations in AI , 2018, FAT.

[12]  Johannes Griss,et al.  Digital image analysis improves precision of PD‐L1 scoring in cutaneous melanoma , 2018, Histopathology.

[13]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[14]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Andrea Vedaldi,et al.  Interpretable Explanations of Black Boxes by Meaningful Perturbation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[17]  Harmanpreet Kaur,et al.  Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning , 2020, CHI.

[18]  Mohamed Zaki,et al.  High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach , 2018, ICML.

[19]  Benjamin Van Roy,et al.  Deep Exploration via Bootstrapped DQN , 2016, NIPS.

[20]  Percy Liang,et al.  Understanding Black-box Predictions via Influence Functions , 2017, ICML.

[21]  Morten Fjeld,et al.  Scale Stain: Multi-Resolution Feature Enhancement in Pathology Visualization , 2016, ArXiv.

[22]  Jianxu Chen,et al.  Automatic Lymphocyte Detection in H&E Images with Deep Neural Networks , 2016, ArXiv.

[23]  Horst K. Hahn,et al.  Deep learning nuclei detection: A simple approach can deliver state-of-the-art results , 2018, Comput. Medical Imaging Graph..

[24]  Georg Langs,et al.  Causability and explainability of artificial intelligence in medicine , 2019, WIREs Data Mining Knowl. Discov..

[25]  Bram van Ginneken,et al.  Automated Gleason Grading of Prostate Biopsies using Deep Learning , 2019, ArXiv.

[26]  Dustin Tran,et al.  Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors , 2018, ArXiv.

[27]  Charles DeCarli,et al.  Interpretable classification of Alzheimer’s disease pathologies with a convolutional neural network pipeline , 2018 .

[28]  Mohammad Emtiyaz Khan,et al.  Approximate Inference Turns Deep Networks into Gaussian Processes , 2019, NeurIPS.

[29]  Morten Fjeld,et al.  Understanding Design for Automated Image Analysis in Digital Pathology , 2016, NordiCHI.

[30]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[31]  Amina Adadi,et al.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) , 2018, IEEE Access.

[32]  Luis Cano,et al.  Automatic Lymphocyte Detection on Gastric Cancer IHC Images Using Deep Learning , 2017, 2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS).

[33]  Zhe L. Lin,et al.  Top-Down Neural Attention by Excitation Backprop , 2016, International Journal of Computer Vision.

[34]  Alexander Binder,et al.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation , 2015, PloS one.

[35]  Claes Lundström,et al.  A Closer Look at Domain Shift for Deep Learning in Histopathology , 2019, ArXiv.

[36]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[37]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[38]  Harris Papadopoulos,et al.  Conformal Prediction with Neural Networks , 2007, 19th IEEE International Conference on Tools with Artificial Intelligence(ICTAI 2007).

[39]  Martin Wattenberg,et al.  Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making , 2019, CHI.

[40]  Martin Wattenberg,et al.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) , 2017, ICML.

[41]  Yarin Gal,et al.  Real Time Image Saliency for Black Box Classifiers , 2017, NIPS.

[42]  Francesco Ciompi,et al.  Deep learning assisted mitotic counting for breast cancer , 2019, Laboratory Investigation.

[43]  Bolei Zhou,et al.  Network Dissection: Quantifying Interpretability of Deep Visual Representations , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Thomas Brox,et al.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks , 2016, NIPS.

[45]  Tommi S. Jaakkola,et al.  A causal framework for explaining the predictions of black-box sequence-to-sequence models , 2017, EMNLP.

[46]  Been Kim,et al.  Sanity Checks for Saliency Maps , 2018, NeurIPS.

[47]  Alexander Binder,et al.  Explaining nonlinear classification decisions with deep Taylor decomposition , 2015, Pattern Recognit..

[48]  David Barber,et al.  A Scalable Laplace Approximation for Neural Networks , 2018, ICLR.

[49]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[50]  David Lopez-Paz,et al.  Single-Model Uncertainties for Deep Learning , 2018, NeurIPS.

[51]  Joel H. Saltz,et al.  Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[53]  Arindam Ghosh,et al.  High-Magnification Multi-views Based Classification of Breast Fine Needle Aspiration Cytology Cell Samples Using Fusion of Decisions from Deep Convolutional Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  William J. Clancey,et al.  Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI , 2019, ArXiv.

[55]  Geert J. S. Litjens,et al.  Learning to detect lymphocytes in immunohistochemistry with deep learning , 2019, Medical Image Anal..

[56]  Marcus Liwicki,et al.  A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference , 2019, ArXiv.

[57]  Daniel Kifer,et al.  Evaluating aleatoric and epistemic uncertainties of time series deep learning models for soil moisture predictions , 2019, ArXiv.

[58]  Jesper Molin,et al.  Implementation of large-scale routine diagnostics using whole slide imaging in Sweden: Digital pathology experiences 2006-2013 , 2014, Journal of pathology informatics.

[59]  Wataru Shimoda,et al.  Distinct Class Saliency Maps for Multiple Object Images , 2016 .

[60]  Yongxiang Huang,et al.  CELNet: Evidence Localization for Pathology Images using Weakly Supervised Learning , 2019, MICCAI.

[61]  Freddy Lécué,et al.  Explainable AI: The New 42? , 2018, CD-MAKE.

[62]  Finale Doshi-Velez,et al.  Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning , 2017, ICML.

[63]  Philipp Berens,et al.  Test-time Data Augmentation for Estimation of Heteroscedastic Aleatoric Uncertainty in Deep Neural Networks , 2018 .

[64]  Alex J. Cannon,et al.  Non-crossing nonlinear regression quantiles by monotone composite quantile regression neural network, with application to rainfall extremes , 2018, Stochastic Environmental Research and Risk Assessment.

[65]  Bolei Zhou,et al.  Object Detectors Emerge in Deep Scene CNNs , 2014, ICLR.

[66]  Jason Yosinski,et al.  Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks , 2016, ArXiv.

[67]  Phedias Diamandis,et al.  Visualizing histopathologic deep learning classification and anomaly detection using nonlinear feature space dimensionality reduction , 2018, BMC Bioinformatics.

[68]  George M. Whitesides,et al.  Self-assembled monolayers of alkanethiols on gold: the adsorption and wetting properties of monolayers derived from two components with alkane chains of different lengths , 1992 .

[69]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[70]  Filip Karlo Dosilovic,et al.  Explainable artificial intelligence: A survey , 2018, 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO).

[71]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[72]  Klaus-Robert Müller,et al.  Learning how to explain neural networks: PatternNet and PatternAttribution , 2017, ICLR.

[73]  Matt Fredrikson,et al.  Influence-Directed Explanations for Deep Convolutional Networks , 2018, 2018 IEEE International Test Conference (ITC).

[74]  Iam Palatnik de Sousa,et al.  Local Interpretable Model-Agnostic Explanations for Classification of Lymph Node Metastases , 2019, Sensors.

[75]  Quanshi Zhang,et al.  Interpretable Convolutional Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[76]  Thomas Brox,et al.  Inverting Visual Representations with Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Andreas Holzinger,et al.  Towards a Deeper Understanding of How a Pathologist Makes a Diagnosis: Visualization of the Diagnostic Process in Histopathology , 2019, 2019 IEEE Symposium on Computers and Communications (ISCC).

[78]  Francois Fleuret,et al.  Full-Gradient Representation for Neural Network Visualization , 2019, NeurIPS.

[79]  Ryan P. Adams,et al.  Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks , 2015, ICML.

[80]  B. van Ginneken,et al.  Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study. , 2020, The Lancet. Oncology.

[81]  Sebastian Nowozin,et al.  DISCO Nets : DISsimilarity COefficients Networks , 2016, NIPS.

[82]  Ellery Wulczyn,et al.  Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer , 2018, npj Digital Medicine.

[83]  A. Madabhushi,et al.  Artificial intelligence in digital pathology — new tools for diagnosis and precision oncology , 2019, Nature Reviews Clinical Oncology.

[84]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[85]  Hao Chen,et al.  Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks , 2016, AAAI.

[86]  Kimmo Kartasalo,et al.  Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study. , 2020, The Lancet. Oncology.

[87]  Julien Cornebise,et al.  Weight Uncertainty in Neural Network , 2015, ICML.

[88]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[89]  Xiaoming Liu,et al.  Do Convolutional Neural Networks Learn Class Hierarchy? , 2017, IEEE Transactions on Visualization and Computer Graphics.

[90]  Nasir M. Rajpoot,et al.  Uncertainty Driven Pooling Network for Microvessel Segmentation in Routine Histology Images , 2018, COMPAY/OMIA@MICCAI.

[91]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[92]  Alexander Binder,et al.  Evaluating the Visualization of What a Deep Neural Network Has Learned , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[93]  Hammad Qureshi,et al.  Translational AI and Deep Learning in Diagnostic Pathology , 2019, Front. Med..

[94]  Yang Zhang,et al.  A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations , 2018, ICML.

[95]  Aleksey Boyko,et al.  Detecting Cancer Metastases on Gigapixel Pathology Images , 2017, ArXiv.

[96]  Arvind Satyanarayan,et al.  The Building Blocks of Interpretability , 2018 .

[97]  Minsuk Kahng,et al.  Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers , 2018, IEEE Transactions on Visualization and Computer Graphics.

[98]  Myunghee Cho Paik,et al.  Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation , 2020, Comput. Stat. Data Anal..

[99]  Nicolai Meinshausen,et al.  Quantile Regression Forests , 2006, J. Mach. Learn. Res..

[100]  Zoubin Ghahramani,et al.  Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference , 2015, ArXiv.

[101]  Laith Alzubaidi,et al.  A Robust Deep Learning Approachto Detect Nuclei in Histopathological Images , 2017 .

[102]  Tim Pearce,et al.  Uncertainty in Neural Networks: Approximately Bayesian Ensembling , 2018, AISTATS.

[103]  Daisuke Komura,et al.  Machine Learning Methods for Histopathological Image Analysis , 2017, Computational and structural biotechnology journal.

[104]  Shan-e-Ahmed Raza,et al.  DeepSDCS: Dissecting cancer proliferation heterogeneity in Ki67 digital whole slide images , 2018, ArXiv.

[105]  H. Cai,et al.  An Affine Invariant Region Detector Using the 4th Differential Invariant , 2007 .

[106]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[107]  Jaewoo Kang,et al.  An automatic nuclei segmentation method based on deep convolutional neural networks for histopathology images , 2019, BMC Biomedical Engineering.

[108]  Cuntai Guan,et al.  A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[109]  Federico Tombari,et al.  Sampling-Free Epistemic Uncertainty Estimation Using Approximated Variance Propagation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[110]  Richard L. Smith,et al.  Essentials of Statistical Inference: Index , 2005 .

[111]  Didrik Nielsen,et al.  Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam , 2018, ICML.

[112]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[113]  Karl Rohr,et al.  Predicting breast tumor proliferation from whole‐slide images: The TUPAC16 challenge , 2018, Medical Image Anal..

[114]  Max Welling,et al.  Visualizing Deep Neural Network Decisions: Prediction Difference Analysis , 2017, ICLR.

[115]  Andrea Vedaldi,et al.  Visualizing Deep Convolutional Neural Networks Using Natural Pre-images , 2015, International Journal of Computer Vision.

[116]  A. Kiureghian,et al.  Aleatory or epistemic? Does it matter? , 2009 .

[117]  Quanshi Zhang,et al.  Visual interpretability for deep learning: a survey , 2018, Frontiers of Information Technology & Electronic Engineering.

[118]  Patrick D. McDaniel,et al.  Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning , 2018, ArXiv.

[119]  Andreas Holzinger,et al.  Measuring the Quality of Explanations: The System Causability Scale (SCS) , 2020, KI - Künstliche Intelligenz.

[120]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[121]  Martin Wattenberg,et al.  SmoothGrad: removing noise by adding noise , 2017, ArXiv.

[122]  Alexander Binder,et al.  Unmasking Clever Hans predictors and assessing what machines really learn , 2019, Nature Communications.

[123]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[124]  Timo Ropinski,et al.  Classifying the classifier: dissecting the weight space of neural networks , 2020, ECAI.

[125]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[126]  Olaf Hellwich,et al.  Deep convolutional neural networks for automatic classification of gastric carcinoma using whole slide images in digital histopathology , 2017, Comput. Medical Imaging Graph..

[127]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[128]  Kevin Smith,et al.  Bayesian Uncertainty Estimation for Batch Normalized Deep Networks , 2018, ICML.

[129]  Tianfu Wu,et al.  Towards Interpretable Object Detection by Unfolding Latent Structures , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[130]  Chandan Chakraborty,et al.  Her2Net: A Deep Framework for Semantic Segmentation and Classification of Cell Membranes and Nuclei in Breast Cancer Evaluation , 2018, IEEE Transactions on Image Processing.