Estimating and abstracting the 3D structure of feline bones using neural networks on X-ray (2D) images

Computing 3D bone models using traditional Computed Tomography (CT) requires a high-radiation dose, cost and time. We present a fully automated, domain-agnostic method for estimating the 3D structure of a bone from a pair of 2D X-ray images. Our triplet loss-trained neural network extracts a 128-dimensional embedding of the 2D X-ray images. A classifier then finds the most closely matching 3D bone shape from a predefined set of shapes. Our predictions have an average root mean square (RMS) distance of 1.08 mm between the predicted and true shapes, making our approach more accurate than the average achieved by eight other examined 3D bone reconstruction approaches. Each embedding extracted from a 2D bone image is optimized to uniquely identify the 3D bone CT from which the 2D image originated and can serve as a kind of fingerprint of each bone; possible applications include faster, image content-based bone database searches for forensic purposes. Jana Čavojská et al. develop a deep learning method to estimate bone 3D structure from 2D X-ray images using abstraction-based classification. Their method is automated, agnostic to previous knowledge about bone geometry, and may be useful in forensic applications.

[1]  Yifan Gong,et al.  Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[2]  Nir Ailon,et al.  Deep Metric Learning Using Triplet Network , 2014, SIMBAD.

[3]  Vittorio Ferrari,et al.  Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading , 2019, International Journal of Computer Vision.

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yann LeCun,et al.  Dimensionality Reduction by Learning an Invariant Mapping , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  P J Prendergast,et al.  A method to reconstruct patient-specific proximal femur surface models from planar pre-operative radiographs. , 2010, Medical engineering & physics.

[7]  Bhallamudi Ravi,et al.  3D femur model reconstruction from biplane X-ray images: a novel method based on Laplacian surface deformation , 2015, International Journal of Computer Assisted Radiology and Surgery.

[8]  Manohar Paluri,et al.  Metric Learning with Adaptive Density Discrimination , 2015, ICLR.

[9]  Gustavo Carneiro,et al.  Learning Local Image Descriptors with Deep Siamese and Triplet Convolutional Networks by Minimizing Global Loss Functions , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  F. Boas,et al.  CT artifacts: Causes and reduction techniques , 2012 .

[11]  Abhinav Gupta,et al.  Marr Revisited: 2D-3D Alignment via Surface Normal Prediction , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Paolo Cignoni,et al.  MeshLab: an Open-Source Mesh Processing Tool , 2008, Eurographics Italian Chapter Conference.

[13]  Ian D. Reid,et al.  Fast Training of Triplet-Based Deep Binary Embedding Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Lucas Beyer,et al.  In Defense of the Triplet Loss for Person Re-Identification , 2017, ArXiv.

[15]  Patrice Delmas,et al.  3D Reconstruction of Patient Specific Bone Models from 2D Radiographs for Image Guided Orthopedic Surgery , 2009, 2009 Digital Image Computing: Techniques and Applications.

[16]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Aren Jansen,et al.  CNN architectures for large-scale audio classification , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[19]  Simant Prakoonwit.  Towards Multiple 3D Bone Surface Identification and Reconstruction Using Few 2D X-Ray Images for Intraoperative Applications , 2014, Int. J. Art Cult. Des. Technol..

[20]  Nanning Zheng,et al.  Person Re-identification by Multi-Channel Parts-Based CNN with Improved Triplet Loss Function , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  C. Lee Giles,et al.  Learning a Hierarchical Latent-Variable Model of 3D Shapes , 2017, 2018 International Conference on 3D Vision (3DV).

[22]  Timo Ropinski,et al.  Single‐image Tomography: 3D Volumes from 2D Cranial X‐Rays , 2017, Comput. Graph. Forum.

[23]  John P. Collomosse,et al.  Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network , 2017, Comput. Vis. Image Underst..

[24]  Ioannis A. Kakadiaris,et al.  End-to-End 3D Face Reconstruction with Deep Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Dong Liu,et al.  Multi-Scale Triplet CNN for Person Re-Identification , 2016, ACM Multimedia.

[26]  Stéphane Lavallée,et al.  Nonrigid 3-D/2-D Registration of Images Using Statistical Models , 1999, MICCAI.

[27]  Song Bai,et al.  Triplet-Center Loss for Multi-view 3D Object Retrieval , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[29]  Zhonglin Zhu,et al.  Construction of 3D human distal femoral surface models using a 3D statistical deformable model. , 2011, Journal of biomechanics.

[30]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  William Ching,et al.  Patient-based radiographic exposure factor selection: a systematic review , 2014, Journal of medical radiation sciences.

[32]  Jitendra Malik,et al.  Category-specific object reconstruction from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Mohammed Bennamoun,et al.  Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[35]  D Mitton,et al.  A Biplanar Reconstruction Method Based on 2D and 3D Contours: Application to the Distal Femur , 2003, Computer methods in biomechanics and biomedical engineering.

[36]  Randy E. Ellis,et al.  2D/3D Deformable Registration Using a Hybrid Atlas , 2005, MICCAI.

[37]  Paolo Cignoni,et al.  Metro: Measuring Error on Simplified Surfaces , 1998, Comput. Graph. Forum.

[38]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Touradj Ebrahimi,et al.  MESH: measuring errors between surfaces using the Hausdorff distance , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[40]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[41]  Marleen de Bruijne,et al.  2D-3D shape reconstruction of the distal femur from stereo X-ray imaging using statistical shape models , 2011, Medical Image Anal..

[42]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[43]  Shengping Zhang,et al.  Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Michael H. F. Wilkinson,et al.  Reproducibility of Standing Posture for X-Ray Radiography: A Feasibility Study of the BalancAid with Healthy Young Subjects , 2010, Annals of Biomedical Engineering.

[45]  Teruhisa Misu,et al.  Boosting Standard Classification Architectures Through a Ranking Regularizer , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[46]  Stefano Filippi,et al.  Analysis of existing methods for 3D modelling of femurs starting from two orthogonal images and development of a script for a commercial software package , 2008, Comput. Methods Programs Biomed..