Modality Bank: Learn multi-modality images across data centers without sharing medical data

Multi-modality images have been widely used and provide comprehensive information for medical image analysis. However, acquiring all modalities among all institutes is costly and often impossible in clinical settings. To leverage more comprehensive multi-modality information, we propose privacy secured decentralized multi-modality adaptive learning architecture named ModalityBank. Our method could learn a set of effective domain-specific modulation parameters plugged into a common domain-agnostic network. We demonstrate by switching different sets of configurations, the generator could output high-quality images for a specific modality. Our method could also complete the missing modalities across all data centers, thus could be used for modality completion purposes. The downstream task trained from the synthesized multi-modality samples could achieve higher performance than learning from one real data center and achieve close-to-real performance compare with all real images.

[1]  Bernd J Pichler,et al.  Multimodal imaging approaches: PET/CT and PET/MRI. , 2008, Handbook of experimental pharmacology.

[2]  G. Annas.  HIPAA regulations - a new era of medical-record privacy? , 2003, The New England journal of medicine.

[3]  Hubert Eichner,et al.  Towards Federated Learning at Scale: System Design , 2019, SysML.

[4]  S. Thacker,et al.  HIPAA privacy rule and public health. Guidance from CDC and the U.S. Department of Health and Human Services. , 2003, MMWR supplements.

[5]  P. Wust,et al.  Comparison of CT, MRI and FDG-PET in response prediction of patients with locally advanced rectal cancer after multimodal preoperative therapy: Is there a benefit in using functional imaging? , 2005, European Radiology.

[6]  Dimitris N. Metaxas,et al.  Synthetic Learning: Learn From Distributed Asynchronized Discriminator GAN Without Sharing Medical Image Data , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[8]  Peter Richtárik,et al.  Federated Learning: Strategies for Improving Communication Efficiency , 2016, ArXiv.

[9]  F. Carrilho,et al.  Multimodality Screening of Hepatic Nodules in Patients With Congenital Heart Disease After Fontan Procedure: Role of Ultrasound, ARFI Elastography, CT, and MRI. , 2018, AJR. American journal of roentgenology.

[10]  M. Wintermark,et al.  Multimodal CT in stroke imaging: new concepts. , 2009, Radiologic clinics of North America.

[11]  Yinghuan Shi,et al.  Deep Symmetric Adaptation Network for Cross-Modality Medical Image Segmentation , 2021, IEEE Transactions on Medical Imaging.

[12]  Yikai Zhang,et al.  Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information , 2020, ArXiv.

[13]  Sijia Wang,et al.  GAN Memory with No Forgetting , 2020, NeurIPS.

[14]  Li Zhang,et al.  Deep similarity learning for multimodal medical images , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[15]  Yu Chen,et al.  OctopusNet: A Deep Learning Segmentation Network for Multi-modal Medical Images , 2019, MMMI@MICCAI.

[16]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Christos Davatzikos,et al.  Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features , 2017, Scientific Data.

[18]  Agustí Verde Parera,et al.  General data protection regulation , 2018 .

[19]  Marion Smits,et al.  Consensus recommendations for a standardized Brain Tumor Imaging Protocol in clinical trials. , 2015, Neuro-oncology.

[20]  Robert M. Nishikawa,et al.  Cross-Organ, Cross-Modality Transfer Learning: Feasibility Study for Segmentation and Classification , 2020, IEEE Access.

[21]  Nannan Li,et al.  MRI Cross-Modality Image-to-Image Translation , 2020, Scientific Reports.

[22]  Daguang Xu,et al.  Privacy-preserving Federated Brain Tumour Segmentation , 2019, MLMI@MICCAI.

[23]  Brian B. Avants,et al.  The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) , 2015, IEEE Transactions on Medical Imaging.

[24]  Hao Chen,et al.  Unsupervised Cross-Modality Domain Adaptation of ConvNets for Biomedical Image Segmentations with Adversarial Loss , 2018, IJCAI.

[25]  Wei Shi,et al.  Federated learning of predictive models from federated Electronic Health Records , 2018, Int. J. Medical Informatics.

[26]  Ramesh Raskar,et al.  FedML: A Research Library and Benchmark for Federated Machine Learning , 2020, ArXiv.

[27]  Yikai Zhang,et al.  Learn distributed GAN with Temporary Discriminators , 2020, ECCV.

[28]  M. Goddard.  The EU General Data Protection Regulation (GDPR): European Regulation that has a Global Impact , 2017 .

[29]  Andrew D. Brown,et al.  Using machine learning for sequence-level automated MRI protocol selection in neuroradiology , 2018, J. Am. Medical Informatics Assoc..

[30]  Hao Chen,et al.  Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation , 2019, AAAI.

[31]  Pedro M. Domingos.  A few useful things to know about machine learning , 2012, Commun. ACM.