AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem?

With the unprecedented developments in deep learning, automatic segmentation of main abdominal organs seems to be a solved problem as state-of-the-art (SOTA) methods have achieved comparable results with inter-rater variability on many benchmark datasets. However, most of the existing abdominal datasets only contain single-center, single-phase, single-vendor, or single-disease cases, and it is unclear whether the excellent performance can generalize on diverse datasets. This paper presents a large and diverse abdominal CT organ segmentation dataset, termed AbdomenCT-1K, with more than 1000 (1K) CT scans from 12 medical centers, including multi-phase, multi-vendor, and multi-disease cases. Furthermore, we conduct a large-scale study for liver, kidney, spleen, and pancreas segmentation and reveal the unsolved segmentation problems of the SOTA methods, such as the limited generalization ability on distinct medical centers, phases, and unseen diseases. To advance the unsolved problems, we further build four organ segmentation benchmarks for fully supervised, semi-supervised, weakly supervised, and continual learning, which are currently challenging and active research topics. Accordingly, we develop a simple and effective method for each benchmark, which can be used as out-of-the-box methods and strong baselines. We believe the AbdomenCT-1K dataset will promote future in-depth research towards clinical applicable abdominal organ segmentation methods.

[1]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[2]  Hai Huang,et al.  Precision Liver Resection: Three-Dimensional Reconstruction Combined with Fluorescence Laparoscopic Imaging , 2020, Surgical innovation.

[3]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[4]  Fredrik Kahl,et al.  Robust Abdominal Organ Segmentation Using Regional Convolutional Neural Networks , 2017, SCIA.

[5]  Yang Wen,et al.  Weakly Supervised Learning of Recurrent Residual ConvNets for Pancreas Segmentation in CT Scans , 2019, 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[6]  Suchi Saria,et al.  Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist , 2020, Nature Medicine.

[7]  Bram van Ginneken,et al.  Fully Automatic Volume Measurement of the Spleen at CT Using Deep Learning. , 2020, Radiology. Artificial intelligence.

[8]  Yan Huang,et al.  Box-Driven Class-Wise Region Masking and Filling Rate Guided Loss for Weakly Supervised Semantic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  J. Mongan,et al.  Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. , 2020, Radiology. Artificial intelligence.

[10]  Yunchao Wei,et al.  Weakly Supervised Scene Parsing with Point-based Distance Metric Learning , 2018, AAAI.

[11]  Liang Zhang,et al.  Block Level Skip Connections Across Cascaded V-Net for Multi-Organ Segmentation , 2020, IEEE Transactions on Medical Imaging.

[12]  Adam P. Harrison,et al.  Organ at Risk Segmentation for Head and Neck Cancer Using Stratified Learning and Neural Architecture Search , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Hao Chen,et al.  The Liver Tumor Segmentation Benchmark (LiTS) , 2019, Medical Image Anal..

[14]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[15]  Ben Glocker,et al.  Joint Supervoxel Classification Forest for Weakly-Supervised Organ Segmentation , 2017, MLMI@MICCAI.

[16]  Yoshua Bengio,et al.  An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks , 2013, ICLR.

[17]  Xinjian Chen,et al.  Automatic Liver Segmentation Based on Shape Constraints and Deformable Graph Cut in CT Images , 2015, IEEE Transactions on Image Processing.

[18]  Xin Yang,et al.  Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved? , 2018, IEEE Transactions on Medical Imaging.

[19]  Jian Sun,et al.  ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Daguang Xu,et al.  DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  L. Joskowicz,et al.  Inter-observer variability of manual contour delineation of structures in CT , 2018, European Radiology.

[22]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[23]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[24]  Stefan Wermter,et al.  Continual Lifelong Learning with Neural Networks: A Review , 2018, Neural Networks.

[25]  Adam P. Harrison,et al.  Pathological Pulmonary Lobe Segmentation from CT Images Using Progressive Holistically Nested Neural Networks and Random Walker , 2017, DLMIA/ML-CDS@MICCAI.

[26]  Luc Van Gool,et al.  A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Daniel L. Rubin,et al.  CT-ORG, a new dataset for multiple organ segmentation in computed tomography , 2020, Scientific Data.

[28]  Ming-Ming Cheng,et al.  Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Ronald M. Summers,et al.  Spatial Aggregation of Holistically-Nested Networks for Automated Pancreas Segmentation , 2016, MICCAI.

[30]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[31]  Bennett A. Landman,et al.  Semi-Supervised Multi-Organ Segmentation through Quality Assurance Supervision , 2019, Medical Imaging: Image Processing.

[32]  Jun Ma,et al.  Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike? , 2021, ArXiv.

[33]  Ronald M. Summers,et al.  Spatial aggregation of holistically‐nested convolutional neural networks for automated pancreas localization and segmentation☆ , 2017, Medical Image Anal..

[34]  Marius George Linguraru,et al.  Abdominal multi-organ segmentation from CT images using conditional shape-location and unsupervised intensity priors , 2015, Medical Image Anal..

[35]  H. Fujita,et al.  Three-Dimensional CT Image Segmentation by Combining 2 D Fully Convolutional Network with 3 D Majority Voting , 2016 .

[36]  Quoc V. Le,et al.  Self-Training With Noisy Student Improves ImageNet Classification , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Martin Styner,et al.  Comparison and Evaluation of Methods for Liver Segmentation From CT Datasets , 2009, IEEE Transactions on Medical Imaging.

[38]  Luc Van Gool,et al.  The 2017 DAVIS Challenge on Video Object Segmentation , 2017, ArXiv.

[39]  Xiaoping Yang,et al.  Learning Geodesic Active Contours for Embedding Object Global Information in Segmentation CNNs , 2020, IEEE Transactions on Medical Imaging.

[40]  Daniel Rueckert,et al.  Discriminative dictionary learning for abdominal multi-organ segmentation , 2015, Medical Image Anal..

[41]  Benedikt Pfülb,et al.  A comprehensive, application-oriented study of catastrophic forgetting in DNNs , 2019, ICLR.

[42]  Tobias Gass,et al.  Cloud-Based Evaluation of Anatomical Structure Segmentation and Landmark Detection Algorithms: VISCERAL Anatomy Benchmarks , 2016, IEEE Transactions on Medical Imaging.

[43]  Ronald M. Summers,et al.  Segmentation label propagation using deep convolutional neural networks and dense conditional random field , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[44]  Andreas Nürnberger,et al.  CHAOS Challenge - Combined (CT-MR) Healthy Abdominal Organ Segmentation , 2020, Medical Image Anal..

[45]  Dean C. Barratt,et al.  Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks , 2018, IEEE Transactions on Medical Imaging.

[46]  Lei Xing,et al.  Modified U-Net (mU-Net) With Incorporation of Object-Dependent High Level Features for Improved Liver and Liver-Tumor Segmentation in CT Images , 2019, IEEE Transactions on Medical Imaging.

[47]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[48]  Ronald M. Summers,et al.  A Review of Deep Learning in Medical Imaging: Imaging Traits, Technology Trends, Case Studies With Progress Highlights, and Future Promises , 2020, Proceedings of the IEEE.

[49]  Stephen M. Moore,et al.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository , 2013, Journal of Digital Imaging.

[50]  Matthias De Lange,et al.  Continual learning: A comparative study on how to defy forgetting in classification tasks , 2019, ArXiv.

[51]  Nima Tajbakhsh,et al.  Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation , 2019, Medical Image Anal..

[52]  Andriy Myronenko,et al.  3D MRI brain tumor segmentation using autoencoder regularization , 2018, BrainLes@MICCAI.

[53]  Ronald M. Summers,et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms , 2019, ArXiv.

[54]  Hongbo Zhang,et al.  3D liver segmentation using multiple region appearances and graph cuts. , 2015, Medical physics.

[55]  Yan Wang,et al.  Recurrent Saliency Transformation Network for Tiny Target Segmentation in Abdominal CT Scans , 2020, IEEE Transactions on Medical Imaging.

[56]  Wei Shen,et al.  Semi-Supervised 3D Abdominal Multi-Organ Segmentation Via Deep Multi-Planar Co-Training , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[57]  Geraint Rees,et al.  Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy , 2018, ArXiv.

[58]  S. Kevin Zhou,et al.  Marginal loss and exclusion loss for partially supervised multi-organ segmentation , 2020, Medical Image Anal..

[59]  Huimin Ma,et al.  Weakly-Supervised Semantic Segmentation by Iterative Affinity Learning , 2020, International Journal of Computer Vision.

[60]  Nikolaos Papanikolopoulos,et al.  An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in CT imaging. , 2020 .

[61]  Feng Chen,et al.  Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets , 2016, International Journal of Computer Assisted Radiology and Surgery.

[62]  B. van Ginneken,et al.  Computer-aided diagnosis: how to move from the laboratory to the clinic. , 2011, Radiology.

[63]  Yong Fan,et al.  A deep learning model integrating FCNNs and CRFs for brain tumor segmentation , 2017, Medical Image Anal..

[64]  Davide Maltoni,et al.  CORe50: a New Dataset and Benchmark for Continuous Object Recognition , 2017, CoRL.

[65]  Yaozong Gao,et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge , 2019, Medical Image Anal..

[66]  Song Wang,et al.  Three-Dimensional CT Image Segmentation by Combining 2D Fully Convolutional Network with 3D Majority Voting , 2016, LABELS/DLMIA@MICCAI.

[67]  Dong Yang,et al.  Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation , 2020, Medical Image Anal..

[68]  Yan Wang,et al.  Abdominal multi-organ segmentation with organ-attention networks and statistical fusion , 2018, Medical Image Anal..

[69]  Trevor Darrell,et al.  Constrained Convolutional Neural Networks for Weakly Supervised Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[70]  Yan Wang,et al.  A Fixed-Point Model for Pancreas Segmentation in Abdominal CT Scans , 2016, MICCAI.

[71]  Ashish Verma,et al.  An Automated System for Atlas Based Multiple OrganSegmentation of Abdominal CT Images , 2016 .

[72]  Yuichiro Hayashi,et al.  An application of cascaded 3 D fully convolutional networks for medical image segmentation , 2018 .

[73]  Klaus H. Maier-Hein,et al.  Automated Design of Deep Learning Methods for Biomedical Image Segmentation , 2019 .

[74]  Holger H. Hoos,et al.  A survey on semi-supervised learning , 2019, Machine Learning.

[75]  Josien P. W. Pluim,et al.  Not‐so‐supervised: A survey of semi‐supervised, multi‐instance, and transfer learning in medical image analysis , 2018, Medical Image Anal..

[76]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[77]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[78]  Ronald M. Summers,et al.  DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation , 2015, MICCAI.

[79]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[80]  Fei-Fei Li,et al.  What's the Point: Semantic Segmentation with Point Supervision , 2015, ECCV.

[81]  Yang Zhang,et al.  Fully convolutional neural network with post-processing methods for automatic liver segmentation from CT , 2017, 2017 Chinese Automation Congress (CAC).

[82]  Xiangrong Zhou,et al.  Construction of a probabilistic atlas for automated liver segmentation in non-contrast torso CT images , 2005 .

[83]  Jie Li,et al.  A sparse annotation strategy based on attention-guided active learning for 3D medical image segmentation , 2019, ArXiv.

[84]  Seong-Whan Lee,et al.  Cascaded MultiTask 3-D Fully Convolutional Networks for Pancreas Segmentation , 2019, IEEE Transactions on Cybernetics.

[85]  Yuichiro Hayashi,et al.  A multi-scale pyramid of 3D fully convolutional networks for abdominal multi-organ segmentation , 2018, MICCAI.

[86]  Luc Van Gool,et al.  The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation , 2019, ArXiv.

[87]  Daguang Xu,et al.  V-NAS: Neural Architecture Search for Volumetric Medical Image Segmentation , 2019, 2019 International Conference on 3D Vision (3DV).

[88]  Maxwell D. Collins,et al.  Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation , 2020, ArXiv.

[89]  Adam P. Harrison,et al.  DeepTarget: Gross tumor and clinical target volume segmentation in esophageal cancer radiotherapy , 2020, Medical Image Anal..

[90]  Yuichiro Hayashi,et al.  An application of cascaded 3D fully convolutional networks for medical image segmentation , 2018, Comput. Medical Imaging Graph..

[91]  Adam P. Harrison,et al.  Co-Heterogeneous and Adaptive Segmentation from Multi-Source and Multi-Phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation , 2020, ECCV.

[92]  Xing Zhang,et al.  Automatic Liver Segmentation Using a Statistical Shape Model With Optimal Surface Detection , 2010, IEEE Transactions on Biomedical Engineering.

[93]  Yan Shen,et al.  Scribble-Based Hierarchical Weakly Supervised Learning for Brain Tumor Segmentation , 2019, MICCAI.

[94]  Mrityunjaya V. Latte,et al.  Combined endeavor of Neutrosophic Set and Chan-Vese model to extract accurate liver image from CT scan , 2017, Comput. Methods Programs Biomed..

[95]  Stefan Wermter,et al.  Continual Lifelong Learning with Neural Networks: A Review , 2019, Neural Networks.

[96]  Seyed-Ahmad Ahmadi,et al.  Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields , 2016, MICCAI.

[97]  Nassim Bouteldja,et al.  OBELISK‐Net: Fewer layers to solve 3D multi‐organ segmentation with sparse deformable convolutions , 2019, Medical Image Anal..

[98]  Yannis Avrithis,et al.  Label Propagation for Deep Semi-Supervised Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[99]  Anne L. Martel,et al.  Loss odyssey in medical image segmentation , 2021, Medical Image Anal..

[100]  Xinlei Chen,et al.  Prior-Aware Neural Network for Partially-Supervised Multi-Organ Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[101]  Derek Hoiem,et al.  Learning without Forgetting , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[102]  Mert R. Sabuncu,et al.  Multi-atlas segmentation of biomedical images: A survey , 2014, Medical Image Anal..

[103]  Hans-Peter Meinzer,et al.  Computerized planning of liver surgery - an overview , 2002, Comput. Graph..

[104]  Bennett A Landman,et al.  Efficient multi-atlas abdominal segmentation on clinically acquired CT with SIMPLE context learning , 2015, Medical Image Anal..

[105]  Jonathan Sykes,et al.  Reflections on the current status of commercial automated segmentation systems in clinical practice , 2014, Journal of medical radiation sciences.

[106]  Giorgio Metta,et al.  Incremental robot learning of new objects with fixed update time , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).