XraySyn: Realistic View Synthesis From a Single Radiograph Through CT Priors

A radiograph visualizes the internal anatomy of a patient through the use of X-ray, which projects 3D information onto a 2D plane. Hence, radiograph analysis naturally requires physicians to relate the prior about 3D human anatomy to 2D radiographs. Synthesizing novel radiographic views in a small range can assist physicians in interpreting anatomy more reliably; however, radiograph view synthesis is heavily ill-posed, lacking in paired data, and lacking in differentiable operations to leverage learning-based approaches. To address these problems, we use Computed Tomography (CT) for radiograph simulation and design a differentiable projection algorithm, which enables us to achieve geometrically consistent transformations between the radiography and CT domains. Our method, XraySyn, can synthesize novel views on real radiographs through a combination of realistic simulation and finetuning on real radiographs. To the best of our knowledge, this is the first work on radiograph view synthesis. We show that by gaining an understanding of radiography in 3D space, our method can be applied to radiograph bone extraction and suppression without groundtruth bone labels.

[1]  J. H. Hubbell,et al.  Tables of X-Ray Mass Attenuation Coefficients and Mass Energy-Absorption Coefficients 1 keV to 20 MeV for Elements Z = 1 to 92 and 48 Additional Substances of Dosimetric Interest , 1995 .

[2]  W. Kalender,et al.  Correlation between CT numbers and tissue parameters needed for Monte Carlo simulations of clinical dose distributions , 2000 .

[3]  T. Bortfeld,et al.  Correlation between CT numbers and tissue parameters needed for Monte Carlo simulations of clinical dose distributions. , 2000, Physics in medicine and biology.

[4]  Rolf Clackdoyle,et al.  CHAPTER 20 – Analytic Image Reconstruction Methods , 2004 .

[5]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[6]  L. Plank,et al.  Dual-energy X-ray absorptiometry and body composition , 2005, Current opinion in clinical nutrition and metabolic care.

[7]  Aldo Badano,et al.  Accelerating Monte Carlo simulations of photon transport in a voxelized geometry using a massively parallel graphics processing unit. , 2009, Medical physics.

[8]  Richard C. Pais,et al.  The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. , 2011, Medical physics.

[9]  G. Liney,et al.  A method to produce and validate a digitally reconstructed radiograph-based computer simulation for optimisation of chest radiographs acquired with a computed radiography imaging system. , 2011, The British journal of radiology.

[10]  A. Sisniega,et al.  Monte Carlo study of the effects of system geometry and antiscatter grids on cone-beam CT scatter distributions. , 2013, Medical physics.

[11]  M. Visser,et al.  The Health, Aging,body Composition Study Dual-Energy X-Ray Absorptiometry and Body , 2013 .

[12]  Joshua B. Tenenbaum,et al.  Deep Convolutional Inverse Graphics Network , 2015, NIPS.

[13]  Pieter Abbeel,et al.  InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets , 2016, NIPS.

[14]  Abhinav Gupta,et al.  Learning a Predictable and Generative Vector Representation for Objects , 2016, ECCV.

[15]  Jitendra Malik,et al.  View Synthesis by Appearance Flow , 2016, ECCV.

[16]  Leo Joskowicz,et al.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016 , 2016, Lecture Notes in Computer Science.

[17]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[18]  Thomas Brox,et al.  Multi-view 3D Models from Single Images with a Convolutional Network , 2015, ECCV.

[19]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[20]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[21]  Nassir Navab,et al.  X-Ray In-Depth Decomposition: Revealing the Latent Structures , 2016, MICCAI.

[22]  Ersin Yumer,et al.  Transformation-Grounded Image Generation Network for Novel 3D View Synthesis , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Hayit Greenspan,et al.  Lung Structures Enhancement in Chest Radiographs via CT Based FCNN Training , 2018, RAMBO+BIA+TIA@MICCAI.

[24]  Raúl San José Estépar,et al.  Emphysema quantification on simulated X-rays through deep learning techniques , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[25]  Ning Zhang,et al.  Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence , 2018, ECCV.

[26]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Nassir Navab,et al.  DeepDRR - A Catalyst for Machine Learning in Fluoroscopy-guided Procedures , 2018, MICCAI.

[28]  Mathieu Aubry,et al.  AtlasNet: A Papier-M\^ach\'e Approach to Learning 3D Surface Generation , 2018, CVPR 2018.

[29]  Timo Ropinski,et al.  Single‐image Tomography: 3D Volumes from 2D Cranial X‐Rays , 2017, Comput. Graph. Forum.

[30]  Danail Stoyanov,et al.  Image Analysis for Moving Organ, Breast, and Thoracic Images: Third International Workshop, RAMBO 2018, Fourth International Workshop, BIA 2018, and First International Workshop, TIA 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16 and 20, 2018, Proceedings , 2018, Lecture Notes in Computer Science.

[31]  Kai Ma,et al.  X2CT-GAN: Reconstructing CT From Biplanar X-Rays With Generative Adversarial Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Duygu Ceylan,et al.  DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction , 2019, NeurIPS.

[33]  Jitendra Malik,et al.  Mesh R-CNN , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Oral-3D: Reconstructing the 3D Bone Structure of Oral Cavity from 2D Panoramic X-ray , 2020, ArXiv.

[35]  Jingjing Lu,et al.  High-Resolution Chest X-Ray Bone Suppression Using Unpaired CT Structural Priors , 2020, IEEE Transactions on Medical Imaging.

[36]  Adam P. Harrison,et al.  Anatomy-Aware Siamese Network: Exploiting Semantic Asymmetry for Accurate Pelvic Fracture Detection in X-ray Images , 2020, ECCV.

[37]  Yinda Zhang,et al.  DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Yun Liu,et al.  Rethinking Computer-Aided Tuberculosis Diagnosis , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Z. Weng,et al.  X2CT-GAN: Reconstructing CT From Biplanar X-Rays With Generative Adversarial Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).