Aggregating Deep Convolutional Features for Melanoma Recognition in Dermoscopy Images

We present a novel framework for automated melanoma recognition in dermoscorpy images, which is a quite challenging task due to the high intra-class and low inter-class variations between melanoma and non-melanoma (benign). The proposed framework shares merits of deep learning method and local descriptors encoding strategy. Specifically, the deep representations of a dermoscopy image are first extracted using a very deep residual neural network pre-trained on ImageNet. Then these local deep descriptors are aggregated by fisher vector (FV) encoding to build a holistic image representation. Finally, the encoded representations are classified using SVM. In contrast to previous studies with complex preprocessing and feature engineering or directly using existing deep learning architectures with fine-tuning on the skin datasets, our solution is simpler, more compact and capable of producing more discriminative features. Extensive experiments performed on ISBI 2016 Skin lesion challenge dataset corroborate the effectiveness of the proposed method, outperforming state-of-the-art approaches in all evaluation metrics.