Segment Anything in Medical Images

Segment anything model (SAM) has revolutionized natural image segmentation, but its performance on medical images is limited. This work presents MedSAM, the first attempt at extending the success of SAM to medical images, with the goal of creating a universal tool for the segmentation of various medical targets. Specifically, we first curate a large-scale medical image dataset, encompassing over 200,000 masks across 11 different modalities. Then, we develop a simple fine-tuning method to adapt SAM to general medical image segmentation. Comprehensive experiments on 21 3D segmentation tasks and 9 2D segmentation tasks demonstrate that MedSAM outperforms the default SAM model with an average Dice Similarity Coefficient (DSC) of 22.5% and 17.6% on 3D and 2D segmentation tasks, respectively. The code and trained model are publicly available at \url{https://github.com/bowang-lab/MedSAM}.

[1]  Y. Zhang,et al.  Segment Anything Model for Medical Image Analysis: an Experimental Study , 2023, Medical Image Anal..

[2]  Yangming Ou,et al.  Accuracy of Segment-Anything Model (SAM) in medical image segmentation tasks , 2023, ArXiv.

[3]  Xinde Li,et al.  When SAM Meets Medical Images: An Investigation of Segment Anything Model (SAM) on Multi-phase Liver Tumor Segmentation , 2023, ArXiv.

[4]  X. Bai,et al.  Learning to "Segment Anything" in Thermal Infrared Images through Knowledge Distillation with a Large Scale Dataset SATIR , 2023, ArXiv.

[5]  Yizhe Zhang,et al.  Can SAM Segment Polyps? , 2023, ArXiv.

[6]  Yong Jae Lee,et al.  Segment Everything Everywhere All at Once , 2023, ArXiv.

[7]  Li Cheng,et al.  Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications , 2023, Machine Intelligence Research.

[8]  L. Gool,et al.  SAM Struggles in Concealed Scenes - Empirical Study on "Segment Anything" , 2023, ArXiv.

[9]  M. Armand,et al.  SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM , 2023, ArXiv.

[10]  Saikat Roy,et al.  SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model , 2023, ArXiv.

[11]  G. Schlaug,et al.  SAM vs BET: A Comparative Study for Brain Extraction and Segmentation of Magnetic Resonance Images using Deep Learning , 2023, ArXiv.

[12]  Bo Li,et al.  Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection , 2023, ArXiv.

[13]  Chunhua Shen,et al.  SegGPT: Segmenting Everything In Context , 2023, ArXiv.

[14]  Ross B. Girshick,et al.  Segment Anything , 2023, 2023 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  M. Riegler,et al.  A multi-centre polyp detection and segmentation dataset for generalisability assessment , 2021, Scientific Data.

[16]  Hao Chen,et al.  The Liver Tumor Segmentation Benchmark (LiTS) , 2019, Medical Image Anal..

[17]  D. Štern,et al.  Fast and Low-GPU-memory abdomen CT organ segmentation: The FLARE challenge , 2022, Medical Image Anal..

[18]  Ping Luo,et al.  AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation , 2022, NeurIPS.

[19]  Anne L. Martel,et al.  Metrics reloaded: Recommendations for image analysis validation , 2022, 2206.01653.

[20]  Ross B. Girshick,et al.  Masked Autoencoders Are Scalable Vision Learners , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Congcong Wang,et al.  AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem? , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Sergio Escalera,et al.  Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge , 2021, IEEE Transactions on Medical Imaging.

[23]  Anne L. Martel,et al.  Loss odyssey in medical image segmentation , 2021, Medical Image Anal..

[24]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[25]  Tom Vercauteren,et al.  Image Compositing for Segmentation of Surgical Tools Without Manual Annotations , 2021, IEEE Transactions on Medical Imaging.

[26]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[27]  L. Giancardo,et al.  Novel Autosegmentation Spatial Similarity Metrics Capture the Time Required to Correct Segmentations Better Than Traditional Metrics in a Thoracic Cavity Segmentation Workflow , 2020, Journal of Digital Imaging.

[28]  Yaozong Gao,et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge , 2019, Medical Image Anal..

[29]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[30]  Jonathan T. Barron,et al.  Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains , 2020, NeurIPS.

[31]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[32]  Zhiqiang He,et al.  Towards Data-Efficient Learning: A Benchmark for COVID-19 CT Lung and Infection Segmentation. , 2020, Medical physics.

[33]  T. Bortfeld,et al.  Anatomical Brain Barriers to Cancer Spread: Segmentation from CT and MR Images , 2020 .

[34]  Walid Al-Dhabyani,et al.  Dataset of breast ultrasound images , 2019, Data in brief.

[35]  Vincent Andrearczyk,et al.  Overview of the HECKTOR Challenge at MICCAI 2020: Automatic Head and Neck Tumor Segmentation in PET/CT , 2020, HECKTOR@MICCAI.

[36]  Emmanuel Iarussi,et al.  Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs , 2020, International Journal of Computer Assisted Radiology and Surgery.

[37]  Ronald M. Summers,et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms , 2019, ArXiv.

[38]  Sébastien Ourselin,et al.  DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[40]  Xin Yang,et al.  Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved? , 2018, IEEE Transactions on Medical Imaging.

[41]  Sébastien Ourselin,et al.  Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning , 2017, IEEE Transactions on Medical Imaging.

[42]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[43]  Hao Chen,et al.  Gland segmentation in colon histology images: The glas challenge contest , 2016, Medical Image Anal..

[44]  Nasir M. Rajpoot,et al.  A Stochastic Polygons Model for Glandular Structures in Colon Histology Images , 2015, IEEE Transactions on Medical Imaging.

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Florian Jung,et al.  Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge , 2014, Medical Image Anal..

[47]  Stephen M. Moore,et al.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository , 2013, Journal of Digital Imaging.

[48]  L. Schwartz,et al.  New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). , 2009, European journal of cancer.

[49]  K. Doi,et al.  Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists' detection of pulmonary nodules. , 2000, AJR. American journal of roentgenology.