Investigating conditional GAN performance with different generator architectures, an ensemble model, and different MR scanners for MR-sCT conversion

PURPOSE Recent developments in MR to synthetic CT (sCT) conversion have shown that treatment planning is possible without an initial planning CT. Promising conversion results have been demonstrated recently using conditional Generative Adversarial Networks (cGANs). However, the performance is generally only tested on images from one MR scanner, which neglects the potential of neural networks to find general high-level abstract features. In this study, we explored the generalizability of the generator models, trained on a single field strength scanner, to data acquired with higher field strengths. METHODS T2-weighted 0.35T MRIs and CTs from 51 patients treated for prostate (40) and cervical cancer (11) were included. 25 of them were used to train four different generators (SE-ResNet, DenseNet, U-Net, and Embedded Net). Further, an ensemble model was created from the four network outputs. The models were validated on 16 patients from a 0.35T MR scanner. Further, the trained models were tested on the Gold Atlas dataset, containing T2-weighted MR scans of different field strengths; 1,5T(7) and 3T(12), and 10 patients from the 0.35T scanner. The sCTs were dosimetrically compared using clinical VMAT plans for all test patients. RESULTS For the same scanner (0.35T), the results from the different models were comparable on the test set, with only minor differences in the MAE (35-51HU body). Similar results were obtained for conversions of 3T GE Signa and the 3T GE Discovery images (40-62HU MAE) for three of the models. However, larger differences were observed for the 1.5T images (48-65HU MAE). The overall best model was found to be the ensemble model. All dose differences were below 1%. CONCLUSION This study shows that it is possible to generalize models trained on images of one scanner to other scanners and different field strengths. Best metric results were achieved by the combination of all networks.

[1]  Di Yan,et al.  Technical Note: U‐net‐generated synthetic CT images for magnetic resonance imaging‐only prostate intensity‐modulated radiation therapy treatment planning , 2018, Medical physics.

[2]  Alex Lallement,et al.  Survey on deep learning for radiotherapy , 2018, Comput. Biol. Medicine.

[3]  Vedat Verter,et al.  Predicting the Need for Ct Imaging in Children with Minor Head Injury Using an Ensemble of Naive Bayes Classifiers , 2022 .

[4]  Fang-Fang Yin,et al.  An Ensemble Approach to Knowledge-Based Intensity-Modulated Radiation Therapy Planning , 2018, Front. Oncol..

[5]  Max A. Viergever,et al.  elastix: A Toolbox for Intensity-Based Medical Image Registration , 2010, IEEE Transactions on Medical Imaging.

[6]  Qian Wang,et al.  Deep embedding convolutional neural network for synthesizing CT image from T1‐Weighted MR image , 2018, Medical Image Anal..

[7]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Quoc V. Le,et al.  Do Better ImageNet Models Transfer Better? , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Michael G. Endres,et al.  Use of Crowd Innovation to Develop an Artificial Intelligence–Based Solution for Radiation Therapy Targeting , 2019, JAMA oncology.

[10]  Lior Rokach,et al.  Ensemble learning: A survey , 2018, WIREs Data Mining Knowl. Discov..

[11]  Fredrik Nordström,et al.  MR-OPERA: A Multicenter/Multivendor Validation of Magnetic Resonance Imaging-Only Prostate Treatment Planning Using Synthetic Computed Tomography Images. , 2017, International journal of radiation oncology, biology, physics.

[12]  Yaozong Gao,et al.  Estimating CT Image From MRI Data Using Structured Random Forest and Auto-Context Model , 2016, IEEE Transactions on Medical Imaging.

[13]  Ali Madani,et al.  Deep echocardiography: data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease , 2018, npj Digital Medicine.

[14]  Berkman Sahiner,et al.  Deep learning in medical imaging and radiation therapy. , 2018, Medical physics.

[15]  Yaozong Gao,et al.  Estimating CT Image from MRI Data Using 3D Fully Convolutional Networks , 2016, LABELS/DLMIA@MICCAI.

[16]  Ming Dong,et al.  Generating synthetic CTs from magnetic resonance images using generative adversarial networks , 2018, Medical physics.

[17]  Aaron Carass,et al.  Unpaired Brain MR-to-CT Synthesis Using a Structure-Constrained CycleGAN , 2018, DLMIA/ML-CDS@MICCAI.

[18]  Sasa Mutic,et al.  The ViewRay system: magnetic resonance-guided and controlled radiotherapy. , 2014, Seminars in radiation oncology.

[19]  T. Nyholm,et al.  A review of substitute CT generation for MRI-only radiation therapy , 2017, Radiation oncology.

[20]  Margie Hunt,et al.  Dosimetric and workflow evaluation of first commercial synthetic CT software for clinical use in pelvis , 2017, Physics in medicine and biology.

[21]  Dinggang Shen,et al.  Medical Image Synthesis with Deep Convolutional Adversarial Networks , 2018, IEEE Transactions on Biomedical Engineering.

[22]  D. Georg,et al.  Advanced optimization methods for whole pelvic and local prostate external beam therapy. , 2016, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[23]  Jason Dowling,et al.  A review of segmentation and deformable registration methods applied to adaptive cervical cancer radiation therapy treatment planning , 2015, Artif. Intell. Medicine.

[24]  D. Georg,et al.  Attenuation correction of a flat table top for radiation therapy in hybrid PET/MR using CT- and 68Ge/68Ga transmission scan-based μ-maps. , 2019, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[25]  Ciprian Catana,et al.  Deep Convolution Neural Network (DCNN) multi-plane approach to synthetic CT generation from MR images - application in brain proton therapy. , 2019, International journal of radiation oncology, biology, physics.

[26]  J G M Kok,et al.  Integrating a 1.5 T MRI scanner with a 6 MV accelerator: proof of concept , 2009, Physics in medicine and biology.

[27]  Tufve Nyholm,et al.  MR and CT data with multiobserver delineations of organs in the pelvic area—Part of the Gold Atlas project , 2018, Medical physics.

[28]  Peter R Seevinck,et al.  Dose evaluation of fast synthetic-CT generation using a generative adversarial network for general pelvis MR-only radiotherapy , 2018, Physics in medicine and biology.

[29]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[30]  Jelmer M. Wolterink,et al.  MR-Only Brain Radiation Therapy: Dosimetric Evaluation of Synthetic CTs Generated by a Dilated Convolutional Neural Network. , 2018, International journal of radiation oncology, biology, physics.

[31]  S. Ourselin,et al.  Evaluation of a multi-atlas CT synthesis approach for MRI-only radiotherapy treatment planning , 2017, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[32]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Fredrik Nordström,et al.  Technical Note: MRI only prostate radiotherapy planning using the statistical decomposition algorithm. , 2015, Medical physics.

[34]  Gang Sun,et al.  Squeeze-and-Excitation Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Jelmer M. Wolterink,et al.  Deep MR to CT Synthesis Using Unpaired Data , 2017, SASHIMI@MICCAI.

[36]  Habib Zaidi,et al.  Atlas-guided generation of pseudo-CT images for MRI-only and hybrid PET–MRI-guided radiotherapy treatment planning , 2016, Physics in medicine and biology.

[37]  Xiao Han,et al.  MR‐based synthetic CT generation using a deep convolutional neural network method , 2017, Medical physics.

[38]  Jeffrey A Fessler,et al.  Female pelvic synthetic CT generation based on joint intensity and shape analysis. , 2017, Physics in medicine and biology.

[39]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Stefan Klein,et al.  Fast parallel image registration on CPU and GPU for diagnostic classification of Alzheimer's disease , 2013, Front. Neuroinform..

[41]  A N T J Kotte,et al.  First patients treated with a 1.5 T MRI-Linac: clinical proof of concept of a high-precision, high-field MRI guided radiotherapy treatment , 2017, Physics in Medicine and Biology.

[42]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[43]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).