Generative models: an upcoming innovation in musculoskeletal radiology? A preliminary test in spine imaging

BackgroundDeep learning is a ground-breaking technology that is revolutionising many research and industrial fields. Generative models are recently gaining interest. Here, we investigate their potential, namely conditional generative adversarial networks, in the field of magnetic resonance imaging (MRI) of the spine, by performing clinically relevant benchmark cases.MethodsFirst, the enhancement of the resolution of T2-weighted (T2W) images (super-resolution) was tested. Then, automated image-to-image translation was tested in the following tasks: (1) from T1-weighted to T2W images of the lumbar spine and (2) vice versa; (3) from T2W to short time inversion-recovery (STIR) images; (4) from T2W to turbo inversion recovery magnitude (TIRM) images; (5) from sagittal standing x-ray projections to T2W images. Clinical and quantitative assessments of the outputs by means of image quality metrics were performed. The training of the models was performed on MRI and x-ray images from 989 patients.ResultsThe performance of the models was generally positive and promising, but with several limitations. The number of disc protrusions or herniations showed good concordance (κ = 0.691) between native and super-resolution images. Moderate-to-excellent concordance was found when translating T2W to STIR and TIRM images (κ ≥ 0.842 regarding disc degeneration), while the agreement was poor when translating x-ray to T2W images.ConclusionsConditional generative adversarial networks are able to generate perceptually convincing synthetic images of the spine in super-resolution and image-to-image translation tasks. Taking into account the limitations of the study, deep learning-based generative methods showed the potential to be an upcoming innovation in musculoskeletal radiology.

[1]  Dong Yu,et al.  Deep Learning: Methods and Applications , 2014, Found. Trends Signal Process..

[2]  Vipin Chaudhary,et al.  Disc herniation diagnosis in MRI using a CAD framework and a two-level classifier , 2012, International Journal of Computer Assisted Radiology and Surgery.

[3]  Jurgen Fripp,et al.  Automatic Substitute Computed Tomography Generation and Contouring for Magnetic Resonance Imaging (MRI)-Alone External Beam Radiation Therapy From Standard MRI Sequences. , 2015, International journal of radiation oncology, biology, physics.

[4]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[5]  T J Masaryk,et al.  Degenerative disk disease: assessment of changes in vertebral body marrow with MR imaging. , 1988, Radiology.

[6]  Ruslan Salakhutdinov,et al.  Learning Deep Generative Models , 2009 .

[7]  Eric Van Reeth,et al.  Super-resolution in magnetic resonance imaging: A review , 2012 .

[8]  Andrew Zisserman,et al.  ISSLS PRIZE IN BIOENGINEERING SCIENCE 2017: Automation of reading of radiological features from magnetic resonance images (MRIs) of the lumbar spine without human intervention is comparable with an expert radiologist , 2017, European Spine Journal.

[9]  Justin K Scheer,et al.  Development of a preoperative predictive model for major complications following adult spinal deformity surgery. , 2017, Journal of neurosurgery. Spine.

[10]  Frank Niemeyer,et al.  Exploring the Potential of Generative Adversarial Networks for Synthesizing Radiological Images of the Spine to be Used in In Silico Trials , 2018, Front. Bioeng. Biotechnol..

[11]  Zengchang Qin,et al.  Auto-painter: Cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks , 2018, Neurocomputing.

[12]  N. Matsunaga,et al.  MRI of bone tumors: Fast STIR imaging as a substitute for T1‐weighted contrast‐enhanced fat‐suppressed spin‐echo imaging , 2004, Journal of magnetic resonance imaging : JMRI.

[13]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[14]  Viktor Vegh,et al.  MRI resolution enhancement: How useful are shifted images obtained by changing the demodulation frequency? , 2011, Magnetic resonance in medicine.

[15]  E. Benzel,et al.  Use of artificial neural networks to predict surgical satisfaction in patients with lumbar spinal canal stenosis: clinical article. , 2014, Journal of neurosurgery. Spine.

[16]  Ian J. Goodfellow,et al.  NIPS 2016 Tutorial: Generative Adversarial Networks , 2016, ArXiv.

[17]  Dinggang Shen,et al.  3D conditional generative adversarial networks for high-quality PET image estimation at low dose , 2018, NeuroImage.

[18]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Ole Winther,et al.  Autoencoding beyond pixels using a learned similarity metric , 2015, ICML.

[20]  Jonathan J Wyatt,et al.  Systematic Review of Synthetic Computed Tomography Generation Methodologies for Use in Magnetic Resonance Imaging-Only Radiation Therapy. , 2018, International journal of radiation oncology, biology, physics.

[21]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Max A. Viergever,et al.  Generative Adversarial Networks for Noise Reduction in Low-Dose CT , 2017, IEEE Transactions on Medical Imaging.

[23]  Jun Yu,et al.  Voxel-wise uncertainty in CT substitute derived from MRI. , 2012, Medical physics.

[24]  Justin K Scheer,et al.  Potential of predictive computer models for preoperative patient selection to enhance overall quality-adjusted life years gained at 2-year follow-up: a simulation in 234 patients with adult spinal deformity. , 2017, Neurosurgical focus.

[25]  Jinsoo Uh,et al.  MRI-based treatment planning with pseudo CT generated through atlas registration. , 2014, Medical physics.

[26]  L R Schad,et al.  Radiosurgical treatment planning of brain metastases based on a fast, three-dimensional MR imaging technique. , 1994, Magnetic resonance imaging.

[27]  J. Frahm,et al.  Reply to: MRI resolution enhancement: How useful are shifted images obtained by changing the demodulation frequency? , 2011, Magnetic resonance in medicine.

[28]  S. Aoki,et al.  Automated brain tissue and myelin volumetry based on quantitative MR imaging with various in-plane resolutions. , 2017, Journal of neuroradiology. Journal de neuroradiologie.

[29]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[30]  K. Lovblad,et al.  Normal Values of Magnetic Relaxation Parameters of Spine Components with the Synthetic MRI Sequence , 2018, American Journal of Neuroradiology.

[31]  Lei Xing,et al.  A unifying probabilistic Bayesian approach to derive electron density from MRI for radiation therapy treatment planning , 2014, Physics in medicine and biology.

[32]  Fabio Galbusera,et al.  Artificial neural networks for the recognition of vertebral landmarks in the lumbar spine , 2018, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[33]  Wenjun Zhang,et al.  An improved full-reference image quality metric based on structure compensation , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[34]  Hua Wang,et al.  Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks , 2017, ArXiv.

[35]  M. Gado.  MRI of the brain. , 1987, Current problems in diagnostic radiology.

[36]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[37]  Lina J. Karam,et al.  A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD) , 2011, IEEE Transactions on Image Processing.

[38]  S Peled,et al.  Superresolution in MRI: Application to human white matter fiber tract visualization by diffusion tensor imaging , 2001, Magnetic resonance in medicine.

[39]  M. Langer,et al.  Comparison of turbo inversion recovery magnitude (TIRM) with T2-weighted turbo spin-echo and T1-weighted spin-echo MR imaging in the early diagnosis of acute osteomyelitis in children , 1998, Pediatric Radiology.

[40]  Taghi M. Khoshgoftaar,et al.  Deep learning applications and challenges in big data analytics , 2015, Journal of Big Data.

[41]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.