Frequency-Supervised MR-to-CT Image Synthesis

This paper strives to generate a synthetic computed tomography (CT) image from a magnetic resonance (MR) image. The synthetic CT image is valuable for radiotherapy planning when only an MR image is available. Recent approaches have made large strides in solving this challenging synthesis problem with convolutional neural networks that learn a mapping from MR inputs to CT outputs. In this paper, we find that all existing approaches share a common limitation: reconstruction breaks down in and around the high-frequency parts of CT images. To address this common limitation, we introduce frequencysupervised deep networks to explicitly enhance high-frequency MR-toCT image reconstruction. We propose a frequency decomposition layer that learns to decompose predicted CT outputs into lowand highfrequency components, and we introduce a refinement module to improve high-frequency reconstruction through high-frequency adversarial learning. Experimental results on a new dataset with 45 pairs of 3D MR-CT brain images show the effectiveness and potential of the proposed approach. Code is available at https://github.com/shizenglin/ Frequency-Supervised-MR-to-CT-Image-Synthesis.

[1]  Ming Dong,et al.  Generating synthetic CTs from magnetic resonance images using generative adversarial networks , 2018, Medical physics.

[2]  Mattias P. Heinrich,et al.  Multi-Atlas Based Pseudo-CT Synthesis Using Multimodal Image Registration and Local Atlas Fusion Strategies , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[3]  Xiao Han,et al.  MR‐based synthetic CT generation using a deep convolutional neural network method , 2017, Medical physics.

[4]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[5]  S. Ourselin,et al.  Evaluation of a multi-atlas CT synthesis approach for MRI-only radiotherapy treatment planning , 2017, Physica medica : PM : an international journal devoted to the applications of physics to medicine and biology : official journal of the Italian Association of Biomedical Physics.

[6]  C. Kuhl,et al.  MRI-Based Attenuation Correction for Hybrid PET/MRI Systems: A 4-Class Tissue Segmentation Technique Using a Combined Ultrashort-Echo-Time/Dixon MRI Sequence , 2012, The Journal of Nuclear Medicine.

[7]  Xiang Zhou,et al.  Unpaired Mr to CT Synthesis with Explicit Structural Constrained Adversarial Learning , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[8]  Sébastien Ourselin,et al.  On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task , 2017, IPMI.

[9]  Ninon Burgos,et al.  Joint Segmentation and CT Synthesis for MRI-only Radiotherapy Treatment Planning , 2016, MICCAI.

[10]  Ninon Burgos,et al.  Iterative framework for the joint segmentation and CT synthesis of MR images: application to MRI-only radiotherapy treatment planning , 2017, Physics in medicine and biology.

[11]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[12]  Snehashis Roy,et al.  Synthesizing CT from Ultrashort Echo-Time MR Images via Convolutional Neural Networks , 2017, SASHIMI@MICCAI.

[13]  Ninon Burgos,et al.  Robust CT Synthesis for Radiotherapy Planning: Application to the Head and Neck Region , 2015, MICCAI.

[14]  Yaozong Gao,et al.  Estimating CT Image from MRI Data Using 3D Fully Convolutional Networks , 2016, LABELS/DLMIA@MICCAI.

[15]  Xuenan Cui,et al.  Deep CT to MR Synthesis Using Paired and Unpaired Data , 2018, Sensors.

[16]  Chuyang Ye,et al.  Frequency-Selective Learning for CT to MR Synthesis , 2020, SASHIMI@MICCAI.

[17]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Xiangyu Zhang,et al.  Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  A. McMillan,et al.  Deep learning Mr imaging–based attenuation correction for PeT/Mr imaging 1 , 2017 .

[20]  D Forsberg,et al.  Generating patient specific pseudo-CT of the head from MR using atlas-based regression , 2015, Physics in medicine and biology.

[21]  T N Hangartner,et al.  Thresholding technique for accurate analysis of density and geometry in QCT, pQCT and microCT images. , 2007, Journal of musculoskeletal & neuronal interactions.

[22]  Brian F. Hutton,et al.  Improved MR to CT synthesis for PET/MR attenuation correction using Imitation Learning , 2019, SASHIMI@MICCAI.

[23]  H. Quick,et al.  Magnetic Resonance–Based Attenuation Correction for PET/MR Hybrid Imaging Using Continuous Valued Attenuation Maps , 2013, Investigative radiology.

[24]  Tiina Seppälä,et al.  A dual model HU conversion from MRI intensity values within and outside of bone segment for MRI-based radiotherapy treatment planning of prostate cancer. , 2013, Medical physics.

[25]  Shu Liao,et al.  Unpaired whole-body MR to CT synthesis with correlation coefficient constrained adversarial learning , 2019, Medical Imaging: Image Processing.

[26]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[27]  Alexia Jolicoeur-Martineau,et al.  The relativistic discriminator: a key element missing from standard GAN , 2018, ICLR.

[28]  Renaud de Crevoisier,et al.  Comparison of deep learning-based and patch-based methods for pseudo-CT generation in MRI-based prostate dose planning. , 2019, International journal of radiation oncology, biology, physics.

[29]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[30]  Caroline Lafond,et al.  Pseudo-CT Generation for MRI-Only Radiation Therapy Treatment Planning: Comparison Among Patch-Based, Atlas-Based, and Bulk Density Methods. , 2019, International journal of radiation oncology, biology, physics.