CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans

Human readers or radiologists routinely perform full-body multi-organ multi-disease detection and diagnosis in clinical practice, while most medical AI systems are built to focus on single organs with a narrow list of a few diseases. This might severely limit AI’s clinical adoption. A certain number of AI models need to be assembled nontrivially to match the diagnostic process of a human reading a CT scan. In this paper, we construct a Unified Tumor Transformer (CancerUniT) model to jointly detect tumor existence & location and diagnose tumor characteristics for eight major cancers in CT scans. CancerUniT is a query-based Mask Transformer model with the output of multi-tumor prediction. We decouple the object queries into organ queries, tumor detection queries and tumor diagnosis queries, and further establish hierarchical relationships among the three groups. This clinically-inspired architecture effectively assists inter- and intra-organ representation learning of tumors and facilitates the resolution of these complex, anatomically related multi-organ cancer image reading tasks. CancerUniT is trained end-to-end using a curated large-scale CT images of 10,042 patients including eight major types of cancers and occurring non-cancer tumors (all are pathology-confirmed with 3D tumor masks annotated by radiologists). On the test set of 631 patients, CancerUniT has demonstrated strong performance under a set of clinically relevant evaluation metrics, substantially outperforming both multi-disease methods and an assembly of eight single-organ expert models in tumor detection, segmentation, and diagnosis. This moves one step closer towards a universal high performance cancer screening tool.

[1]  A. Yuille,et al.  The FELIX Project: Deep Networks To Detect Pancreatic Neoplasms , 2022, medRxiv.

[2]  P. Catalano,et al.  Clinical validation of deep learning algorithms for radiotherapy targeting of non-small-cell lung cancer: an observational study , 2022, The Lancet. Digital health.

[3]  Adam P. Harrison,et al.  A flexible three‐dimensional heterophase computed tomography hepatocellular carcinoma detection algorithm for generalizable and practical screening , 2022, Hepatology communications.

[4]  Le Lu,et al.  Deep Learning for Fully Automated Prediction of Overall Survival in Patients Undergoing Resection for Pancreatic Cancer , 2022, Annals of surgery.

[5]  R. Munden,et al.  Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT. , 2022, Radiology.

[6]  P. Pickhardt.  Value-added Opportunistic CT Screening: State of the Art. , 2022, Radiology.

[7]  Zaiyi Liu,et al.  Mining whole-lung information by artificial intelligence for predicting EGFR genotype and targeted therapy response in lung cancer: a multicohort study. , 2022, The Lancet. Digital health.

[8]  Holger Roth,et al.  Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images , 2022, BrainLes@MICCAI.

[9]  A. Schwing,et al.  Masked-attention Mask Transformer for Universal Image Segmentation , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Song Bai,et al.  TransMix: Attend to Mix for Vision Transformers , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Guotai Wang,et al.  WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image , 2021, Medical Image Anal..

[12]  Le Lu,et al.  External and Internal Validation of a Computer Assisted Diagnostic Model for Detecting Multi-Organ Mass Lesions in CT images. , 2021, Chinese medical sciences journal = Chung-kuo i hsueh k'o hsueh tsa chih.

[13]  Bingbing Ni,et al.  Asymmetric 3D Context Fusion for Universal Lesion Detection , 2021, MICCAI.

[14]  Zhongchao Shi,et al.  Modality-aware Mutual Learning for Multi-modal Medical Image Segmentation , 2021, MICCAI.

[15]  Alexander G. Schwing,et al.  Per-Pixel Classification is Not All You Need for Semantic Segmentation , 2021, NeurIPS.

[16]  C. Swanton,et al.  Clinical validation of a targeted methylation-based multi-cancer early detection test using an independent validation set. , 2021, Annals of oncology : official journal of the European Society for Medical Oncology.

[17]  Bjoern H Menze,et al.  The Medical Segmentation Decathlon , 2021, Nature Communications.

[18]  M. Ingrisch,et al.  Reduction of missed thoracic findings in emergency whole-body computed tomography using artificial intelligence assistance. , 2021, Quantitative imaging in medicine and surgery.

[19]  Cordelia Schmid,et al.  Segmenter: Transformer for Semantic Segmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Youbao Tang,et al.  Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss , 2021, MICCAI.

[21]  Christoph Feichtenhofer,et al.  Multiscale Vision Transformers , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[22]  B. van Ginneken,et al.  Artificial intelligence in radiology: 100 commercially available products and their scientific evidence , 2021, European Radiology.

[23]  Chunhua Shen,et al.  CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation , 2021, MICCAI.

[24]  Yan Wang,et al.  TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation , 2021, ArXiv.

[25]  A. Jemal,et al.  Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries , 2021, CA: a cancer journal for clinicians.

[26]  Matthieu Cord,et al.  Training data-efficient image transformers & distillation through attention , 2020, ICML.

[27]  Jing Xiao,et al.  3D Graph Anatomy Geometry-Integrated Network for Pancreatic Mass Segmentation, Diagnosis, and Quantitative Patient Management , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Jens Petersen,et al.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation , 2020, Nature Methods.

[29]  A. Yuille,et al.  MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Chunhua Shen,et al.  DoDNet: Learning to Segment Multi-Organ and Tumors from Multiple Partially Labeled Datasets , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[32]  Bin Li,et al.  Deformable DETR: Deformable Transformers for End-to-End Object Detection , 2020, ICLR.

[33]  Adam P. Harrison,et al.  Learning From Multiple Datasets With Heterogeneous and Partial Labels for Universal Lesion Detection in CT , 2020, IEEE Transactions on Medical Imaging.

[34]  Adam P. Harrison,et al.  Harvesting, Detecting, and Characterizing Liver Lesions from Large-scale Multi-phase CT Data via Deep Dynamic Texture Learning , 2020, ArXiv.

[35]  Nicolas Usunier,et al.  End-to-End Object Detection with Transformers , 2020, ECCV.

[36]  D. Ledbetter,et al.  Feasibility of blood testing combined with PET-CT to screen for cancer and guide intervention , 2020, Science.

[37]  F. Giganti,et al.  Deep learning radiomic nomogram can predict the number of lymph node metastasis in locally advanced gastric cancer: an international multi-center study. , 2020, Annals of oncology : official journal of the European Society for Medical Oncology.

[38]  David S. Melnick,et al.  International evaluation of an AI system for breast cancer screening , 2020, Nature.

[39]  Jianming Liang,et al.  UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation , 2019, IEEE Transactions on Medical Imaging.

[40]  Lei Wu,et al.  Deep Learning Signature Based on Staging CT for Preoperative Prediction of Sentinel Lymph Node Metastasis in Breast Cancer. , 2019, Academic radiology.

[41]  Yaozong Gao,et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge , 2019, Medical Image Anal..

[42]  Bennett A. Landman,et al.  Semi-Supervised Multi-Organ Segmentation through Quality Assurance Supervision , 2019, Medical Imaging: Image Processing.

[43]  Rainer Hofmann-Wellenhof,et al.  A deep learning system for differential diagnosis of skin diseases , 2019, Nature Medicine.

[44]  Youbao Tang,et al.  MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation , 2019, MICCAI.

[45]  Arie E. Kaufman,et al.  Learning Multi-Class Segmentations From Single-Class Datasets , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  G. Corrado,et al.  End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography , 2019, Nature Medicine.

[47]  Xinlei Chen,et al.  Prior-Aware Neural Network for Partially-Supervised Multi-Organ Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Jiapan Guo,et al.  Automatic Pulmonary Nodule Detection in CT Scans Using Convolutional Neural Networks Based on Maximum Intensity Projection , 2019, IEEE Transactions on Medical Imaging.

[49]  Ronald M. Summers,et al.  Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning From Radiology Reports and Label Ontology , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Weidong Cai,et al.  Knowledge-based Collaborative Deep Learning for Benign-Malignant Lung Nodule Classification on Chest CT , 2019, IEEE Transactions on Medical Imaging.

[51]  Raymond Y Huang,et al.  Artificial intelligence in cancer imaging: Clinical challenges and applications , 2019, CA: a cancer journal for clinicians.

[52]  Yuxing Tang,et al.  Uldor: A Universal Lesion Detector For Ct Scans With Pseudo Masks And Hard Negative Example Mining , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[53]  Hao Chen,et al.  The Liver Tumor Segmentation Benchmark (LiTS) , 2019, Medical Image Anal..

[54]  D. Ahlquist.  Universal cancer screening: revolutionary, rational, and realizable , 2018, npj Precision Oncology.

[55]  Geraint Rees,et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease , 2018, Nature Medicine.

[56]  Alan L. Yuille,et al.  Multi-Scale Coarse-to-Fine Segmentation for Screening Pancreatic Ductal Adenocarcinoma , 2018, MICCAI.

[57]  Le Lu,et al.  DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning , 2018, Journal of medical imaging.

[58]  Yan Wang,et al.  Abdominal multi-organ segmentation with organ-attention networks and statistical fusion , 2018, Medical Image Anal..

[59]  Ludmila V. Danilova,et al.  Detection and localization of surgically resectable cancers with a multi-analyte blood test , 2018, Science.

[60]  Ronald M. Summers,et al.  Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-Scale Lesion Database , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[61]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[62]  Vivek Vaidya,et al.  Lung nodule detection in CT using 3D convolutional neural networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[63]  Yanqi Huang,et al.  Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. , 2016, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[64]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[65]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[67]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[68]  A. Sodickson,et al.  Recurrent CT, cumulative radiation exposure, and associated radiation-induced cancer risks from CT of adults. , 2009, Radiology.

[69]  Kunio Doi,et al.  Computer-aided diagnosis in medical imaging: Historical review, current status and future potential , 2007, Comput. Medical Imaging Graph..

[70]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[71]  Ronald M Summers.  Road maps for advancement of radiologic computer-aided detection in the 21st century. , 2003, Radiology.

[72]  Shan Yang,et al.  TotalSegmentator: robust segmentation of 104 anatomical structures in CT images , 2022, ArXiv.

[73]  Xiansheng Hua,et al.  DeepCRC: Colorectum and Colorectal Cancer Segmentation in CT Scans via Deep Colorectal Coordinate Transform , 2022, MICCAI.

[74]  Xiansheng Hua,et al.  Effective Opportunistic Esophageal Cancer Screening Using Noncontrast CT Imaging , 2022, MICCAI.

[75]  Le Lu,et al.  Effective Pancreatic Cancer Screening on Non-contrast CT Scans via Anatomy-Aware Transformers , 2021, MICCAI.

[76]  Pong C. Yuen,et al.  A Segmentation-Assisted Model for Universal Lesion Detection with Partial Labels , 2021, MICCAI.

[77]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[78]  D. Lynch,et al.  The National Lung Screening Trial: overview and study design. , 2011, Radiology.