The Skincare project, an interactive deep learning system for differential diagnosis of malignant skin lesions. Technical Report

A shortage of dermatologists causes long wait times for patients who seek dermatologic care. In addition, the diagnostic accuracy of general practitioners has been reported to be lower than the accuracy of artificial intelligence software. This article describes the Skincare project (H2020, EIT Digital). Contributions include enabling technology for clinical decision support based on interactive machine learning (IML), a reference architecture towards a Digital European Healthcare Infrastructure (also cf. EIT MCPS), technical components for aggregating digitised patient information, and the integration of decision support technology into clinical test-bed environments. However, the main contribution is a diagnostic and decision support system in dermatology for patients and doctors, an interactive deep learning system for differential diagnosis of malignant skin lesions. In this article, we describe its functionalities and the user interfaces to facilitate machine learning from human input. The baseline deep learning system, which delivers state-of-the-art results and the potential to augment general practitioners and even dermatologists, was developed and validated using de-identified cases from a dermatology image data base (ISIC), which has about 20000 cases for development and validation, provided by board-certified dermatologists defining the reference standard for every case. ISIC allows for differential diagnosis, a ranked list of eight diagnoses, that is used to plan treatments in the common setting of diagnostic ambiguity. We give an overall description of the outcome of the Skincare project, and we focus on the steps to support communication and coordination between humans and machine in IML. This is an integral part of the development of future cognitive assistants in the medical domain, and we describe the necessary intelligent user interfaces.

[1]  Daniel Sonntag,et al.  A CNN toolbox for skin cancer classification , 2019, ArXiv.

[2]  Santosh Pandey,et al.  Skin Cancer Diagnostics with an All-Inclusive Smartphone Application , 2019, Symmetry.

[3]  Ghassan Hamarneh,et al.  Melanoma Recognition via Visual Attention , 2019, IPMI.

[4]  Y. Fujisawa,et al.  Deep‐learning‐based, computer‐aided classifier developed with a small dataset of clinical images surpasses board‐certified dermatologists in skin tumour diagnosis , 2018, The British journal of dermatology.

[5]  Kate Saenko,et al.  RISE: Randomized Input Sampling for Explanation of Black-box Models , 2018, BMVC.

[6]  Minsuk Kahng,et al.  Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers , 2018, IEEE Transactions on Visualization and Computer Graphics.

[7]  G. Argenziano,et al.  The impact of dermoscopy on melanoma detection in the practice of dermatologists in Europe: results of a pan‐European survey , 2017, Journal of the European Academy of Dermatology and Venereology : JEADV.

[8]  Wojciech Samek,et al.  Methods for interpreting and understanding deep neural networks , 2017, Digit. Signal Process..

[9]  Tim Miller,et al.  Explanation in Artificial Intelligence: Insights from the Social Sciences , 2017, Artif. Intell..

[10]  Tom Fahey,et al.  Diagnosing malignant melanoma in ambulatory care: a systematic review of clinical prediction rules , 2017, BMJ Open.

[11]  João Manuel R. S. Tavares,et al.  A computational approach for detecting pigmented skin lesions in macroscopic images , 2016, Expert Syst. Appl..

[12]  Ramprasaath R. Selvaraju,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, International Journal of Computer Vision.

[13]  Daniele Nardi,et al.  Skin lesion image segmentation using Delaunay Triangulation for melanoma detection , 2016, Comput. Medical Imaging Graph..

[14]  Sule Yildirim Yayilgan,et al.  Efficient Melanoma Detection Using Texture-Based RSurf Features , 2016, ICIAR.

[15]  João Paulo Papa,et al.  Computational methods for the image segmentation of pigmented skin lesions: A review , 2016, Comput. Methods Programs Biomed..

[16]  Zachary Chase Lipton.  The mythos of model interpretability , 2016, ACM Queue.

[17]  Zhen Ma,et al.  A Novel Approach to Segment Skin Lesions in Dermoscopic Images Based on a Deformable Model , 2016, IEEE Journal of Biomedical and Health Informatics.

[18]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[19]  Jeremy S Bordeaux,et al.  Early detection of melanoma: reviewing the ABCDEs. , 2015, Journal of the American Academy of Dermatology.

[20]  Ngai-Man Cheung,et al.  Early melanoma diagnosis with mobile imaging , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[21]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[22]  Jorge S. Marques,et al.  Two Systems for the Detection of Melanomas in Dermoscopy Images Using Texture and Color Features , 2014, IEEE Systems Journal.

[23]  J. Scharcanski,et al.  Computer Vision Techniques for the Diagnosis of Skin Cancer , 2013 .

[24]  Pedro M. Ferreira,et al.  PH2 - A dermoscopic image database for research and benchmarking , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[25]  Mircea-Florin Vaida,et al.  Active contours: Application to plant recognition , 2012, 2012 IEEE 8th International Conference on Intelligent Computer Communication and Processing.

[26]  Rafael García,et al.  Computerized analysis of pigmented skin lesions: A review , 2012, Artif. Intell. Medicine.

[27]  D. Rigel,et al.  The Evolution of Melanoma Diagnosis: 25 Years Beyond the ABCDs , 2010, CA: a cancer journal for clinicians.

[28]  Célia A. Zorzo Barcelos,et al.  An automatic based nonlinear diffusion equations scheme for skin lesion segmentation , 2009, Appl. Math. Comput..

[29]  Kathy Boutis,et al.  Should a Head-Injured Child Receive a Head CT Scan? A Systematic Review of Clinical Prediction Rules , 2009, Pediatrics.

[30]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  W. Clem Karl,et al.  A Real-Time Algorithm for the Approximation of Level-Set-Based Curve Evolution , 2008, IEEE Transactions on Image Processing.

[32]  T. Mendonca,et al.  Comparison of Segmentation Methods for Automatic Diagnosis of Dermoscopy Images , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[33]  Z. She,et al.  Combination of features from skin pattern and ABCD analysis for lesion classification , 2007, Skin research and technology : official journal of International Society for Bioengineering and the Skin (ISBS) [and] International Society for Digital Imaging of Skin (ISDIS) [and] International Society for Skin Imaging.

[34]  June K Robinson,et al.  Skills training to learn discrimination of ABCDE criteria by those at risk of developing melanoma. , 2006, Archives of dermatology.

[35]  Randy H. Moss,et al.  Automatic lesion boundary detection in dermoscopy images using gradient vector flow snakes , 2005, Skin research and technology : official journal of International Society for Bioengineering and the Skin (ISBS) [and] International Society for Digital Imaging of Skin (ISDIS) [and] International Society for Skin Imaging.

[36]  A. Kopf,et al.  Early diagnosis of cutaneous melanoma: revisiting the ABCD criteria. , 2004, JAMA.

[37]  L. Vese,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[38]  K Wolff,et al.  In vivo epiluminescence microscopy of pigmented skin lesions. I. Pattern analysis of pigmented skin lesions. , 1987, Journal of the American Academy of Dermatology.

[39]  A. Kopf,et al.  Early detection of malignant melanoma: The role of physician examination and self‐examination of the skin , 1985, CA: a cancer journal for clinicians.

[40]  David Brink,et al.  : A Review of the , 2018 .

[41]  Antonio Krüger,et al.  The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1 , 2017, The Handbook of Multimodal-Multisensor Interfaces, Volume 1.

[42]  Alan Geller,et al.  Methods of Melanoma Detection. , 2016, Cancer treatment and research.

[43]  Ramprasaath R. Selvaraju,et al.  Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization , 2016 .

[44]  Vandana Jagtap,et al.  Computer Aided Melanoma Skin Cancer Detection Using Image Processing , 2015 .

[45]  Aurora Sáez,et al.  Pattern Analysis in Dermoscopic Images , 2014 .

[46]  Jia Deng,et al.  A large-scale hierarchical image database , 2009, CVPR 2009.

[47]  Stephen W Dusza,et al.  The CASH (color, architecture, symmetry, and homogeneity) algorithm for dermoscopy. , 2007, Journal of the American Academy of Dermatology.

[48]  M. E. Celebi,et al.  Border detection in dermoscopy images using hybrid thresholding on optimized color channels , 2011, Comput. Medical Imaging Graph..