Review of 2-D/3-D Reconstruction Using Statistical Shape and Intensity Models and X-Ray Image Synthesis: Toward a Unified Framework

Patient-specific three-dimensional (3-D) bone models are useful for a number of clinical applications such as surgery planning, postoperative evaluation, as well as implant and prosthesis design. Two-dimensional-to-3-D (2-D/3-D) reconstruction, also known as model-to-modality or atlas-based 2-D/3-D registration, provides a means of obtaining a 3-D model of a patient's bones from their 2-D radiographs when 3-D imaging modalities are not available. The preferred approach for estimating both shape and density information (that would be present in a patient's computed tomography data) for 2-D/3-D reconstruction makes use of digitally reconstructed radiographs and deformable models in an iterative, non-rigid, intensity-based approach. Based on a large number of state-of-the-art 2-D/3-D bone reconstruction methods, a unified mathematical formulation of the problem is proposed in a common conceptual framework, using unambiguous terminology. In addition, shortcomings, recent adaptations, and persisting challenges are discussed along with insights for future research.

[1]  Agnes Grünerbl,et al.  3D image segmentation using combined shape-intensity prior models , 2007, International Journal of Computer Assisted Radiology and Surgery.

[2]  R. Siddon Fast calculation of the exact radiological path for a three-dimensional CT array. , 1985, Medical physics.

[3]  Max Mignotte,et al.  3D/2D registration and segmentation of scoliotic vertebrae using statistical models. , 2003, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[4]  Simon R. Arridge,et al.  A survey of hierarchical non-linear medical image registration , 1999, Pattern Recognit..

[5]  Guoyan Zheng,et al.  3D reconstruction of a patient-specific surface model of the proximal femur from calibrated x-ray radiographs: a validation study. , 2009, Medical physics.

[6]  J. Jurvelin,et al.  Estimation of 3D shape, internal density and mechanics of proximal femur by combining bone mineral density images with shape and density templates , 2012, Biomechanics and modeling in mechanobiology.

[7]  Wolfgang Birkfellner,et al.  Wobbled splatting—a fast perspective volume rendering method for simulation of x-ray images from CT , 2005, Physics in medicine and biology.

[8]  Cristian Lorenz,et al.  Temporal subtraction of chest radiographs compensating pose differences , 2011, Medical Imaging.

[9]  Wolfgang Birkfellner,et al.  Fast DRR splat rendering using common consumer graphics hardware. , 2007, Medical physics.

[10]  Guoyan Zheng,et al.  Fully automatic segmentation of AP pelvis X-rays via random forest regression with efficient feature selection and hierarchical sparse shape composition , 2014, Comput. Vis. Image Underst..

[11]  Laura Caponetti,et al.  3D Bone Reconstruction From Two X-ray Views , 1990, [1990] Proceedings of the Twelfth Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[12]  Marleen de Bruijne,et al.  2D-3D shape reconstruction of the distal femur from stereo X-ray imaging using statistical shape models , 2011, Medical Image Anal..

[13]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[14]  Thierry Cresson,et al.  3D shape reconstruction of bone from two x-ray images using 2D/3D non-rigid registration based on moving least-squares deformation , 2010, Medical Imaging.

[15]  Wolfgang Birkfellner,et al.  The Zernike Expansion - An Example of a Merit Function for 2D/3D Registration Based on Orthogonal Functions , 2008, MICCAI.

[16]  Wee Kheng Leow,et al.  Recovery of 3D Pose of Bones in Single 2D X-ray Images , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[17]  P. King Medical imaging systems , 1986, Proceedings of the IEEE.

[18]  L. Joskowicz,et al.  Gradient-based 2-D/3-D rigid registration of fluoroscopic X-ray to CT , 2003, IEEE Transactions on Medical Imaging.

[19]  Russell H. Taylor,et al.  Deformable 2D-3D Registration of the Pelvis with a Limited Field of View, Using Shape Statistics , 2007, MICCAI.

[20]  D. Felson,et al.  Magnetic resonance imaging-based three-dimensional bone shape of the knee predicts onset of knee osteoarthritis: data from the osteoarthritis initiative. , 2013, Arthritis and rheumatism.

[21]  J. S. Marron,et al.  Nested Sphere Statistics of Skeletal Models , 2013, Innovations for Shape Analysis, Models and Algorithms.

[22]  Hans-Christian Hege,et al.  Fast Generation of Virtual X-ray Images for Reconstruction of 3D Anatomy , 2013, IEEE Transactions on Visualization and Computer Graphics.

[23]  Iasonas Kokkinos,et al.  3D Model-based Reconstruction of the Proximal Femur from Low-dose Biplanar X-Ray Images , 2011, BMVC.

[24]  Graeme P. Penney,et al.  Fully automated 2D-3D registration and verification , 2015, Medical Image Anal..

[25]  Guoyan Zheng,et al.  Statistical Shape and Deformation Models Based 2D–3D Reconstruction , 2017 .

[26]  Guoyan Zheng,et al.  Scaled, patient-specific 3D vertebral model reconstruction based on 2D lateral fluoroscopy , 2011, International Journal of Computer Assisted Radiology and Surgery.

[27]  Lok Ming Lui,et al.  Landmark constrained genus-one surface Teichmüller map applied to surface registration in medical imaging , 2015, Medical Image Anal..

[28]  Guoyan Zheng,et al.  Effective incorporating spatial information in a mutual information based 3D-2D registration of a CT volume to X-ray images , 2010, Comput. Medical Imaging Graph..

[29]  Andriy Myronenko,et al.  Point Set Registration: Coherent Point Drift , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  F Lavaste,et al.  [Geometrical modeling of the spine and the thorax for the biomechanical analysis of scoliotic deformities using the finite element method]. , 1995, Annales de chirurgie.

[31]  Hans-Christian Hege,et al.  Atlas-based 3D-Shape Reconstruction from X-Ray Images , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[32]  Swapna Banerjee,et al.  2D/3D Non-rigid Image Registration by an Efficient Demons Approach , 2014, 2014 IEEE 27th International Symposium on Computer-Based Medical Systems.

[33]  Lei Wang,et al.  A novel mutual information-based similarity measure for 2D/3D registration in image guided intervention , 2013, 2013 1st International Conference on Orange Technologies (ICOT).

[34]  Eric Nectoux,et al.  Three-dimensional measurements of the lower extremity in children and adolescents using a low-dose biplanar X-ray device , 2012, European Radiology.

[35]  Rüdiger Westermann,et al.  Accelerated volume ray-casting using texture mapping , 2001, Proceedings Visualization, 2001. VIS '01..

[36]  Diego Borro,et al.  Study of a Ray Casting Technique for the Visualization of Deformable Volumes , 2014, IEEE Transactions on Visualization and Computer Graphics.

[37]  David J. Hawkes,et al.  A Comparison of 2D-3D Intensity-Based Registration and Feature-Based Registration for Neurointerventions , 2002, MICCAI.

[38]  Nikos Paragios,et al.  Deformable Medical Image Registration: A Survey , 2013, IEEE Transactions on Medical Imaging.

[39]  D Mitton,et al.  A Biplanar Reconstruction Method Based on 2D and 3D Contours: Application to the Distal Femur , 2003, Computer methods in biomechanics and biomedical engineering.

[40]  Randy E. Ellis,et al.  2D/3D Deformable Registration Using a Hybrid Atlas , 2005, MICCAI.

[41]  J. D. De Guise,et al.  3D reconstruction of the proximal femur with low-dose digital stereoradiography , 2004, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[42]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[43]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[44]  Elena De Momi,et al.  Statistical shape models based 2D/3D registration methods for knee orthopaedic surgery , 2016 .

[45]  Omar Ahmad,et al.  Volumetric DXA (VXA): A new method to extract 3D information from multiple in vivo DXA images , 2010, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[46]  Russell H. Taylor,et al.  Construction and simplification of bone density models , 2001, SPIE Medical Imaging.

[47]  Richard D. Komistek,et al.  A robust method for registration of three-dimensional knee implant models to two-dimensional fluoroscopy images , 2003, IEEE Transactions on Medical Imaging.

[48]  Z. Jane Wang,et al.  A CNN Regression Approach for Real-Time 2D/3D Registration , 2016, IEEE Transactions on Medical Imaging.

[49]  Mauricio Reyes,et al.  Image-based vs. mesh-based statistical appearance models of the human femur: implications for finite element simulations. , 2014, Medical engineering & physics.

[50]  J. Jurvelin,et al.  Assessment of the 3-D shape and mechanics of the proximal femur using a shape template and a bone mineral density image , 2011, Biomechanics and modeling in mechanobiology.

[51]  Kenji Shimada,et al.  Cost‐ and time‐effective three‐dimensional bone‐shape reconstruction from X‐ray images , 2007, The international journal of medical robotics + computer assisted surgery : MRCAS.

[52]  Carolyn Anglin,et al.  Towards Robust Measurement of Pelvic Parameters from AP Radiographs using Articulated 3D Models , 2015 .

[53]  Guoyan Zheng,et al.  Reconstruction of Patient-Specific 3D Bone Model from Biplanar X-Ray Images and Point Distribution Models , 2006, 2006 International Conference on Image Processing.

[54]  P J Prendergast,et al.  A method to reconstruct patient-specific proximal femur surface models from planar pre-operative radiographs. , 2010, Medical engineering & physics.

[55]  Joyoni Dey,et al.  Targeted 2D/3D registration using ray normalization and a hybrid optimizer. , 2006, Medical physics.

[56]  Jürgen Weese,et al.  A comparison of similarity measures for use in 2-D-3-D medical image registration , 1998, IEEE Transactions on Medical Imaging.

[57]  Chengwen Chu,et al.  Fully automatic reconstruction of personalized 3D volumes of the proximal femur from 2D X-ray images , 2016, International Journal of Computer Assisted Radiology and Surgery.

[58]  Guoyan Zheng Personalized X-Ray Reconstruction of the Proximal Femur via Intensity-Based Non-rigid 2D-3D Registration , 2011, MICCAI.

[59]  Daniel Rueckert,et al.  Fast calculation of digitally reconstructed radiographs using light fields , 2003, SPIE Medical Imaging.

[60]  D Mitton,et al.  3D reconstruction of the pelvis from bi-planar radiography , 2006, Computer methods in biomechanics and biomedical engineering.

[61]  Zhiping Mu A Fast DRR Generation Scheme for 3D-2D Image Registration Based on the Block Projection Method , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[62]  Hanna Isaksson,et al.  Generation of 3D shape, density, cortical thickness and finite element mesh of proximal femur from a DXA image , 2015, Medical Image Anal..

[63]  Pavel Zemcík,et al.  Intensity-based femoral atlas 2D/3D registration using Levenberg-Marquardt optimisation , 2016, SPIE Medical Imaging.

[64]  Randy E. Ellis,et al.  Hardware-Assisted 2D/3D Intensity-Based Registration for Assessing Patellar Tracking , 2004, MICCAI.

[65]  Baba C. Vemuri,et al.  Real-Time DRR Generation Using Cylindrical Harmonics , 2002, MICCAI.

[66]  Martin Styner,et al.  Evaluation of 3D Correspondence Methods for Model Building , 2003, IPMI.

[67]  Alejandro F. Frangi,et al.  3D bone mineral density distribution and shape reconstruction of the proximal femur from a single simulated DXA image: an in vitro study , 2010, Medical Imaging.

[68]  P. Hu,et al.  Method for registration of 3D shapes without overlap for known 3D priors , 2021, Electronics Letters.

[69]  W Skalli,et al.  3D reconstruction of the spine from biplanar X-rays using parametric models based on transversal and longitudinal inferences. , 2009, Medical engineering & physics.

[70]  C. Anglin,et al.  Radiological method for measuring patellofemoral tracking and tibiofemoral kinematics before and after total knee replacement , 2012, Bone & joint research.

[71]  Russell H. Taylor,et al.  Statistical Atlases of Bone Anatomy: Construction, Iterative Improvement and Validation , 2007, MICCAI.

[72]  Alejandro F. Frangi,et al.  3D reconstruction of both shape and Bone Mineral Density distribution of the femur from DXA images , 2010, 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[73]  David Mitton,et al.  Fast accurate stereoradiographic 3D-reconstruction of the spine using a combined geometric and statistic model. , 2004, Clinical biomechanics.

[74]  Michel A. Audette,et al.  Statistical Shape Model Construction of Lumbar Vertebrae and Intervertebral Discs in Segmentation for Discectomy Surgery Simulation , 2015, CSI@MICCAI.

[75]  Xun Jia,et al.  Graphics Processing Unit-Based High Performance Computing in Radiation Therapy , 2015 .

[76]  Bhallamudi Ravi,et al.  3D femur model reconstruction from biplane X-ray images: a novel method based on Laplacian surface deformation , 2015, International Journal of Computer Assisted Radiology and Surgery.

[77]  Pietro Cerveri,et al.  2D/3D reconstruction of the distal femur using statistical shape models addressing personalized surgical instruments in knee arthroplasty: A feasibility analysis , 2017, The international journal of medical robotics + computer assisted surgery : MRCAS.

[78]  Robert B. Fisher,et al.  A Comparison of Four Algorithms for Estimating 3-D Rigid Transformations , 1995, BMVC.

[79]  Guoyan Zheng,et al.  Non-rigid free-form 2D-3D registration using a B-spline-based statistical deformation model , 2017, Pattern Recognit..

[80]  Russell H. Taylor,et al.  Integrating Statistical Models of Bone Density into Shape Based 2 D-3 D Registration Framework , 2009 .

[81]  Yuemin Zhu,et al.  Digitally reconstructed radiograph generation by an adaptive Monte Carlo method , 2006, Physics in medicine and biology.

[82]  W. Eric L. Grimson,et al.  2D-3D rigid registration of X-ray fluoroscopy and CT images using mutual information and sparsely sampled histogram estimators , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[83]  Nahmkeon Hur,et al.  The Study of Femoral 3D Reconstruction Process Based on Anatomical Parameters Using a Numerical Method , 2008 .

[84]  Alejandro F. Frangi,et al.  Reconstructing the 3D Shape and Bone Mineral Density Distribution of the Proximal Femur From Dual-Energy X-Ray Absorptiometry , 2011, IEEE Transactions on Medical Imaging.

[85]  Russell H. Taylor,et al.  Simultaneous pose estimation and patient-specific model reconstruction from single image using maximum penalized likelihood estimation (MPLE) , 2016, Pattern Recognit..

[86]  Stephen D. Laycock,et al.  GPU Accelerated Generation of Digitally Reconstructed Radiographs for 2-D/3-D Image Registration , 2012, IEEE Transactions on Biomedical Engineering.

[87]  Wafa Skalli,et al.  3D reconstruction of rib cage geometry from biplanar radiographs using a statistical parametric model approach , 2016, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[88]  Nicholas Ayache,et al.  Articulated Spine Models for 3-D Reconstruction From Partial Radiographic Data , 2008, IEEE Transactions on Biomedical Engineering.

[89]  Frank Sauer,et al.  Automatic registration of portal images and volumetric CT for patient positioning in radiation therapy , 2006, Medical Image Anal..

[90]  Stéphane Lavallée,et al.  Incorporating a statistically based shape model into a system for computer-assisted anterior cruciate ligament surgery , 1999, Medical Image Anal..

[91]  Leo Joskowicz,et al.  Gradient-Based 2D/3D Rigid Registration of Fluoroscopic X-ray to CT , 2003, IEEE Trans. Medical Imaging.

[92]  Max A. Viergever,et al.  Mutual-information-based registration of medical images: a survey , 2003, IEEE Transactions on Medical Imaging.

[93]  M. Wybier,et al.  Musculoskeletal imaging in progress: the EOS imaging system. , 2013, Joint, bone, spine : revue du rhumatisme.

[94]  Tim Cootes,et al.  Detection of vertebral fractures in DXA VFA images using statistical models of appearance and a semi-automatic segmentation , 2010, Osteoporosis International.

[95]  Stefan Klein,et al.  Simultaneous Multiresolution Strategies for Nonrigid Image Registration , 2013, IEEE Transactions on Image Processing.

[96]  M. Levoy,et al.  Fast volume rendering using a shear-warp factorization of the viewing transformation , 1994, SIGGRAPH.

[97]  Farida Cheriet,et al.  A Novel Method for the 3-D Reconstruction of Scoliotic Ribs From Frontal and Lateral Radiographs , 2011, IEEE Transactions on Biomedical Engineering.

[98]  Alejandro F. Frangi,et al.  3D reconstruction of the lumbar vertebrae from anteroposterior and lateral dual-energy X-ray absorptiometry , 2013, Medical Image Anal..

[99]  S. Bierma-Zeinstra,et al.  Total hip replacement but not clinical osteoarthritis can be predicted by the shape of the hip: a prospective cohort study (CHECK). , 2012, Osteoarthritis and cartilage.

[100]  Leo Joskowicz,et al.  Registration of a CT-like atlas to fluoroscopic X-ray images using intensity correspondences , 2008, International Journal of Computer Assisted Radiology and Surgery.

[101]  Zhe Chen,et al.  Automated 2D-3D registration of portal images and CT data using line-segment enhancement. , 2008, Medical physics.

[102]  Alejandro F. Frangi,et al.  Automatic Construction of 3D Statistical Deformation Models Using Non-rigid Registration , 2001, MICCAI.

[103]  A. A. Zadpoor,et al.  Statistical shape and appearance models of bones. , 2014, Bone.

[104]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[105]  Jinwei Sun,et al.  3D Reconstruction Method from Biplanar Radiography Using DLT Algorithm: Application to the Femur , 2010, 2010 First International Conference on Pervasive Computing, Signal Processing and Applications.

[106]  Leo Joskowicz,et al.  Effective Intensity-Based 2D/3D Rigid Registration between Fluoroscopic X-Ray and CT , 2003, MICCAI.

[107]  Marco Viceconti,et al.  Evaluation of the generality and accuracy of a new mesh morphing procedure for the human femur. , 2011, Medical engineering & physics.

[108]  Bjorn De Sutter,et al.  A Fast Algorithm to Calculate the Exact Radiological Path through a Pixel or Voxel Space , 1998 .

[109]  Guoyan Zheng,et al.  Statistical shape model-based reconstruction of a scaled, patient-specific surface model of the pelvis from a single standard AP x-ray radiograph. , 2010, Medical physics.

[110]  R. Aspden,et al.  Femoral geometry as a risk factor for osteoporotic hip fracture in men and women. , 2008, Medical engineering & physics.

[111]  J. Keyak,et al.  Generation of a 3D proximal femur shape from a single projection 2D radiographic image , 2009, Osteoporosis International.

[112]  Kebin Jia,et al.  A GPU-Based DRR Generation Method Using Cubic Window , 2012, 2012 Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing.

[113]  Farida Cheriet,et al.  Personalized X-Ray 3-D Reconstruction of the Scoliotic Spine From Hybrid Statistical and Image-Based Models , 2009, IEEE Transactions on Medical Imaging.

[114]  Zhonglin Zhu,et al.  Construction of 3D human distal femoral surface models using a 3D statistical deformable model. , 2011, Journal of biomechanics.

[115]  Marleen de Bruijne,et al.  Statistical Shape Model-Based Femur Kinematics From Biplane Fluoroscopy , 2012, IEEE Transactions on Medical Imaging.

[116]  David R. Haynor,et al.  PET-CT image registration in the chest using free-form deformations , 2003, IEEE Transactions on Medical Imaging.

[117]  Robert T. Schultz,et al.  Registration of Cortical Anatomical Structures via Robust 3D Point Matching , 1999, IPMI.

[118]  David Staub,et al.  A digitally reconstructed radiograph algorithm calculated from first principles. , 2012, Medical physics.

[119]  Bernhard Schölkopf,et al.  Kernels, regularization and differential equations , 2008, Pattern Recognit..

[120]  Paul A. Yushkevich,et al.  Deformable M-Reps for 3D Medical Image Segmentation , 2003, International Journal of Computer Vision.

[121]  Allan Hanbury,et al.  Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool , 2015, BMC Medical Imaging.

[122]  Takeo Kanade,et al.  Iterative x-ray/ct registration using accelerated volume rendering , 2001 .

[123]  Natalie N. Braun,et al.  Strategies for reducing radiation dose in CT. , 2009, Radiologic clinics of North America.

[124]  Prasanth B. Nair,et al.  Statistical modelling of the whole human femur incorporating geometric and material properties. , 2010, Medical engineering & physics.

[125]  S. Bierma-Zeinstra,et al.  Variation in joint shape of osteoarthritic knees. , 2011, Arthritis and rheumatism.

[126]  Christian Roux,et al.  2-D–3-D Frequency Registration Using a Low-Dose Radiographic System for Knee Motion Estimation , 2013, IEEE Transactions on Biomedical Engineering.

[127]  Hans-Peter Meinzer,et al.  Statistical shape models for 3D medical image segmentation: A review , 2009, Medical Image Anal..

[128]  Tania S. Douglas,et al.  Interactive patient-specific 3D approximation of scapula bone shape from 2D X-ray images using landmark-constrained statistical shape model fitting , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[129]  Raphaël Dumas,et al.  A semi-automated method using interpolation and optimisation for the 3D reconstruction of the spine from bi-planar radiography: a precision and accuracy study , 2007, Medical & Biological Engineering & Computing.

[130]  Thomas Malzbender,et al.  Fourier volume rendering , 1993, TOGS.

[131]  W Skalli,et al.  Fast 3D reconstruction of the lower limb using a parametric model and statistical inferences and clinical measurements calculation from biplanar X-rays , 2012, Computer methods in biomechanics and biomedical engineering.

[132]  E. Ntasis,et al.  Real time digital reconstructed radiograph (DRR) rendering in frequency domain , 1999, Proceedings of the First Joint BMES/EMBS Conference. 1999 IEEE Engineering in Medicine and Biology 21st Annual Conference and the 1999 Annual Fall Meeting of the Biomedical Engineering Society (Cat. N.

[133]  Russell H. Taylor,et al.  Assessing accuracy factors in deformable 2D/3D medical image registration using a statistical pelvis model , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[134]  Cedric Schwartz,et al.  An Automated Statistical Shape Model Developmental Pipeline: Application to the Human Scapula and Humerus , 2015, IEEE Transactions on Biomedical Engineering.

[135]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[136]  Guoyan Zheng 3D volumetric intensity reconsturction from 2D x-ray images using partial least squares regression , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[137]  Johan Thunberg,et al.  Shape‐aware surface reconstruction from sparse 3D point‐clouds , 2016, Medical Image Anal..

[138]  Benedikt Schuler,et al.  Comparison of Different Metrics for Appearance-Model-Based 2D/3D-registration with X-ray Images , 2008, Bildverarbeitung für die Medizin.

[139]  Thomas Gerig,et al.  Gaussian Process Morphable Models , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[140]  Bostjan Likar,et al.  Evaluation of similarity measures for reconstruction-based registration in image-guided radiotherapy and surgery. , 2006, International journal of radiation oncology, biology, physics.