Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation. Thanks to its impressive capabilities in all-round segmentation tasks and its prompt-based interface, SAM has sparked intensive discussion within the community. It is even said by many prestigious experts that image segmentation task has been"finished"by SAM. However, medical image segmentation, although an important branch of the image segmentation family, seems not to be included in the scope of Segmenting"Anything". Many individual experiments and recent studies have shown that SAM performs subpar in medical image segmentation. A natural question is how to find the missing piece of the puzzle to extend the strong segmentation capability of SAM to medical image segmentation. In this paper, instead of fine-tuning the SAM model, we propose Med SAM Adapter, which integrates the medical specific domain knowledge to the segmentation model, by a simple yet effective adaptation technique. Although this work is still one of a few to transfer the popular NLP technique Adapter to computer vision cases, this simple implementation shows surprisingly good performance on medical image segmentation. A medical image adapted SAM, which we have dubbed Medical SAM Adapter (MSA), shows superior performance on 19 medical image segmentation tasks with various image modalities including CT, MRI, ultrasound image, fundus image, and dermoscopic images. MSA outperforms a wide range of state-of-the-art (SOTA) medical image segmentation methods, such as nnUNet, TransUNet, UNetr, MedSegDiff, and also outperforms the fully fine-turned MedSAM with a considerable performance gap. Code will be released at: https://github.com/WuJunde/Medical-SAM-Adapter.

[1]  Bo Wang,et al.  Segment Anything in Medical Images , 2023, ArXiv.

[2]  Saikat Roy,et al.  SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model , 2023, ArXiv.

[3]  Lucas W. Remedios,et al.  Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging , 2023, ArXiv.

[4]  Ross B. Girshick,et al.  Segment Anything , 2023, 2023 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  H. Fu,et al.  Diff-UNet: A Diffusion Embedded Network for Volumetric Segmentation , 2023, ArXiv.

[6]  Yangming Ou,et al.  Accuracy of Segment-Anything Model (SAM) in medical image segmentation tasks , 2023, ArXiv.

[7]  Yehui Yang,et al.  MedSegDiff: Medical Image Segmentation with Diffusion Probabilistic Model , 2022, MIDL.

[8]  H. Greenspan,et al.  RadImageNet: An Open Radiologic Deep Learning Research Dataset for Effective Transfer Learning , 2022, Radiology. Artificial intelligence.

[9]  Ping Luo,et al.  AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation , 2022, NeurIPS.

[10]  Jiangliu Wang,et al.  AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition , 2022, NeurIPS.

[11]  Philippe C. Cattin,et al.  Diffusion Models for Implicit Image Segmentation Ensembles , 2021, MIDL.

[12]  Yoav Goldberg,et al.  BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models , 2021, ACL.

[13]  Yelong Shen,et al.  LoRA: Low-Rank Adaptation of Large Language Models , 2021, ICLR.

[14]  Daguang Xu,et al.  UNETR: Transformers for 3D Medical Image Segmentation , 2021, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[15]  Pengchuan Zhang,et al.  Parameter-efficient Fine-tuning for Vision Transformers , 2022, ArXiv.

[16]  Haidar A. Almubarak,et al.  REFUGE2 Challenge: Treasure for Multi-Domain Learning in Glaucoma Assessment , 2022, ArXiv.

[17]  Holger Roth,et al.  Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images , 2022, BrainLes@MICCAI.

[18]  Lior Wolf,et al.  SegDiff: Image Segmentation with Diffusion Probabilistic Models , 2021, ArXiv.

[19]  Christos Davatzikos,et al.  The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification , 2021, ArXiv.

[20]  Mingzhi Mao,et al.  Multi-Task Learning For Thyroid Nodule Segmentation With Thyroid Region Prior , 2021, 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI).

[21]  Wenxuan Wang,et al.  TransBTS: Multimodal Brain Tumor Segmentation Using Transformer , 2021, MICCAI.

[22]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[23]  Yan Wang,et al.  TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation , 2021, ArXiv.

[24]  Chen Chu,et al.  Ultrasonic thyroid nodule detection method based on U-Net network , 2020, Comput. Methods Programs Biomed..

[25]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[26]  Mohamed Chetoui,et al.  Explainable end-to-end deep learning for diabetic retinopathy detection across multiple datasets , 2020, Journal of medical imaging.

[27]  Zhao Zhang,et al.  Interactive Image Segmentation With First Click Attention , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Xi Fang,et al.  Multi-Organ Segmentation Over Partially Labeled Datasets With Multi-Scale Feature Abstraction , 2020, IEEE Transactions on Medical Imaging.

[29]  Xiaoxiao Li,et al.  REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs , 2019, Medical Image Anal..

[30]  Arcot Sowmya,et al.  Automated Brain Tumor Segmentation Using Multimodal Brain Scans: A Survey Based on Models Submitted to the BraTS 2012–2018 Challenges , 2020, IEEE Reviews in Biomedical Engineering.

[31]  Verónica Vilaplana,et al.  BCN20000: Dermoscopic Lesions in the Wild , 2019, Scientific data.

[32]  Chi-Wing Fu,et al.  Boundary and Entropy-driven Adversarial Learning for Fundus Image Segmentation , 2019, MICCAI.

[33]  Yogesan Kanagasingam,et al.  Robust optic disc and cup segmentation with deep learning for glaucoma detection , 2019, Comput. Medical Imaging Graph..

[34]  Jon Kleinberg,et al.  Transfusion: Understanding Transfer Learning for Medical Imaging , 2019, NeurIPS.

[35]  Md Ashraful Alam Milton.  Automated Skin Lesion Classification Using Ensemble of Deep Neural Networks in ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection Challenge , 2019, ArXiv.

[36]  Yiting Xie,et al.  Pre-training on Grayscale ImageNet Improves Medical Image Classification , 2018, ECCV Workshops.

[37]  Bastian Leibe,et al.  Iteratively Trained Interactive Segmentation , 2018, BMVC.

[38]  Harald Kittler,et al.  The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions , 2018, Scientific Data.

[39]  Noel C. F. Codella,et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC) , 2016, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[40]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.