Segmentation of the Proximal Femur from MR Images using Deep Convolutional Neural Networks

Magnetic resonance imaging (MRI) has been proposed as a complimentary method to measure bone quality and assess fracture risk. However, manual segmentation of MR images of bone is time-consuming, limiting the use of MRI measurements in the clinical practice. The purpose of this paper is to present an automatic proximal femur segmentation method that is based on deep convolutional neural networks (CNNs). This study had institutional review board approval and written informed consent was obtained from all subjects. A dataset of volumetric structural MR images of the proximal femur from 86 subjects were manually-segmented by an expert. We performed experiments by training two different CNN architectures with multiple number of initial feature maps, layers and dilation rates, and tested their segmentation performance against the gold standard of manual segmentations using four-fold cross-validation. Automatic segmentation of the proximal femur using CNNs achieved a high dice similarity score of 0.95 ± 0.02 with precision = 0.95 ± 0.02, and recall = 0.95 ± 0.03. The high segmentation accuracy provided by CNNs has the potential to help bring the use of structural MRI measurements of bone quality into clinical practice for management of osteoporosis.

[1]  S. Majumdar,et al.  Proximal femur: assessment for osteoporosis with T2* decay characteristics at MR imaging. , 1998, Radiology.

[2]  L. R. Dice Measures of the Amount of Ecologic Association Between Species , 1945 .

[3]  Sharmila Majumdar,et al.  Heterogeneity of bone microstructure in the femoral head in patients with osteoporosis: an ex vivo HR-pQCT study. , 2013, Bone.

[4]  Ronald M. Summers,et al.  DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation , 2015, MICCAI.

[5]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  S. Majumdar,et al.  Use of 2D U-Net Convolutional Neural Networks for Automated Cartilage and Meniscus Segmentation of Knee MR Imaging Data to Determine Relaxometry and Morphometry. , 2018, Radiology.

[7]  Vince D. Calhoun,et al.  End-to-end learning of brain tissue segmentation from imperfect labeling , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[8]  S. Majumdar,et al.  Noninvasive assessment of bone mineral and structure: State of the art , 1996, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[9]  Guoyan Zheng,et al.  3D U-net with Multi-level Deep Supervision: Fully Automatic Segmentation of Proximal Femur in 3D MR Images , 2017, MLMI@MICCAI.

[10]  S. Majumdar,et al.  In Vivo Determination of Bone Structure in Postmenopausal Women: A Comparison of HR‐pQCT and High‐Field MR Imaging , 2007, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[11]  S. Majumdar,et al.  Trabecular Bone Architecture in the Distal Radius Using Magnetic Resonance Imaging in Subjects with Fractures of the Proximal Femur , 1999, Osteoporosis International.

[12]  W. Youden,et al.  Index for rating diagnostic tests , 1950, Cancer.

[13]  S Zachow,et al.  Knee menisci segmentation using convolutional neural networks: data from the Osteoarthritis Initiative. , 2018, Osteoarthritis and cartilage.

[14]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[15]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[16]  Hao Chen,et al.  VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images , 2017, NeuroImage.

[17]  Nancy Lane,et al.  Finite Element Analysis of the Proximal Femur and Hip Fracture Risk in Older Men , 2009, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[18]  Patrick van der Smagt,et al.  CNN-based Segmentation of Medical Imaging Data , 2017, ArXiv.

[19]  Shu Liao,et al.  Multi-Instance Deep Learning: Discover Discriminative Local Anatomies for Bodypart Recognition , 2016, IEEE Transactions on Medical Imaging.

[20]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[21]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  W. Skalli,et al.  Volumetric quantitative computed tomography of the proximal femur: relationships linking geometric and densitometric variables to bone strength. Role for compact bone , 2006, Osteoporosis International.

[23]  Nadia Magnenat-Thalmann,et al.  MRI Bone Segmentation Using Deformable Models and Shape Priors , 2008, MICCAI.

[24]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[25]  G. A. Ladinsky,et al.  Trabecular Structure Quantified With the MRI‐Based Virtual Bone Biopsy in Postmenopausal Women Contributes to Vertebral Deformity Burden Independent of Areal Vertebral BMD , 2007, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[26]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[27]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling , 2015, CVPR 2015.

[28]  Seyed-Ahmad Ahmadi,et al.  Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields , 2016, MICCAI.

[29]  Matthew Lai,et al.  Deep Learning for Medical Image Segmentation , 2015, Deep Learning Applications in Medical Imaging.

[30]  Ronald M. Summers,et al.  Active appearance model and deep learning for more accurate prostate segmentation on MRI , 2016, SPIE Medical Imaging.

[31]  Alejandro F. Frangi,et al.  Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015 , 2015, Lecture Notes in Computer Science.

[32]  Julio Carballido-Gamio,et al.  Variable flip angle three‐dimensional fast spin‐echo sequence combined with outer volume suppression for imaging trabecular bone structure of the proximal femur , 2015, Journal of magnetic resonance imaging : JMRI.

[33]  S. Cummings,et al.  Clinical use of bone densitometry: scientific review. , 2002, JAMA.

[34]  Won-Sook Lee,et al.  A 3D active model framework for segmentation of proximal femur in MR images , 2014, International Journal of Computer Assisted Radiology and Surgery.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[37]  Ronald M. Summers,et al.  A New 2.5D Representation for Lymph Node Detection Using Random Sets of Deep Convolutional Neural Network Observations , 2014, MICCAI.

[38]  Nassir Navab,et al.  Preface. The 16th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2013 was held in Nagoya, Japan during September 22-26, 2013. , 2013, International Conference on Medical Image Computing and Computer-Assisted Intervention.

[39]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[40]  Victor Alves,et al.  Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images , 2016, IEEE Transactions on Medical Imaging.

[41]  Thomas M Link,et al.  osteoporosis imaging : State of the Art and Advanced Imaging 1 , 2022 .

[42]  Gregory Chang,et al.  Patient-specific Hip Fracture Strength Assessment with Microstructural MR Imaging-based Finite Element Modeling. , 2017, Radiology.

[43]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[44]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[47]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[48]  Hao Chen,et al.  3D multi‐scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multi‐modality MR Images , 2018, Medical Image Anal..

[49]  Gregory Chang,et al.  Measurement reproducibility of magnetic resonance imaging-based finite element analysis of proximal femur microarchitecture for in vivo assessment of bone strength , 2014, Magnetic Resonance Materials in Physics, Biology and Medicine.

[50]  Steven K Boyd,et al.  Bone strength at the distal radius can be estimated from high-resolution peripheral quantitative computed tomography and the finite element method. , 2008, Bone.

[51]  Terry M Therneau,et al.  Structural patterns of the proximal femur in relation to age and hip fracture risk in women. , 2013, Bone.

[52]  Richard Kijowski,et al.  Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging , 2018, Magnetic resonance in medicine.

[53]  Allan Hanbury,et al.  Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool , 2015, BMC Medical Imaging.

[54]  Le Lu,et al.  Pancreas Segmentation in MRI Using Graph-Based Decision Fusion on Convolutional Neural Networks , 2016, MICCAI.

[55]  Timothy Q. Duong,et al.  Structural and functional MRI reveals multiple retinal layers , 2006, Proceedings of the National Academy of Sciences.

[56]  Punam K. Saha,et al.  Topological analysis of trabecular bone MR images , 2000, IEEE Transactions on Medical Imaging.

[57]  M. Bouxsein,et al.  In vivo assessment of trabecular bone microarchitecture by high-resolution peripheral quantitative computed tomography. , 2005, The Journal of clinical endocrinology and metabolism.

[58]  Makoto Osaki,et al.  Trabecular microfractures in the femoral head with osteoporosis: analysis of microcallus formations by synchrotron radiation micro CT. , 2014, Bone.

[59]  Christian Igel,et al.  Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network , 2013, MICCAI.

[60]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[61]  Joachim Hornegger,et al.  Self-gated Radial MRI for Respiratory Motion Compensation on Hybrid PET/MR Systems , 2013, MICCAI.

[62]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[63]  Sharmila Majumdar,et al.  Magnetic Resonance Imaging of Trabecular Bone Structure , 2002, Topics in magnetic resonance imaging : TMRI.

[64]  Christian Wachinger,et al.  DeepNAT: Deep convolutional neural network for segmenting neuroanatomy , 2017, NeuroImage.

[65]  Lisa Tang,et al.  Deep 3D Convolutional Encoder Networks With Shortcuts for Multiscale Feature Integration Applied to Multiple Sclerosis Lesion Segmentation , 2016, IEEE Transactions on Medical Imaging.

[66]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[67]  Roland Krug,et al.  Feasibility of in vivo structural analysis of high-resolution magnetic resonance images of the proximal femur , 2005, Osteoporosis International.

[68]  P. Rüegsegger,et al.  Direct Three‐Dimensional Morphometric Analysis of Human Cancellous Bone: Microstructural Data from Spine, Femur, Iliac Crest, and Calcaneus , 1999, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[69]  S. Majumdar,et al.  In Vivo High Resolution MRI of the Calcaneus: Differences in Trabecular Structure in Osteoporosis Patients , 1998, Journal of bone and mineral research : the official journal of the American Society for Bone and Mineral Research.

[70]  Steven D. Kugelmass,et al.  Potential role of nuclear magnetic resonance for the evaluation of trabecular bone quality , 2005, Calcified Tissue International.

[71]  Honglak Lee,et al.  A unified framework for automatic wound segmentation and analysis with deep convolutional neural networks , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[72]  Iasonas Kokkinos,et al.  Modeling local and global deformations in Deep Learning: Epitomic convolution, Multiple Instance Learning, and sliding window detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  O. Painter,et al.  Position-squared coupling in a tunable photonic crystal optomechanical cavity , 2015, 1505.07291.

[74]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[75]  William M. Wells,et al.  Medical Image Computing and Computer-Assisted Intervention — MICCAI’98 , 1998, Lecture Notes in Computer Science.

[76]  Jeremy F Magland,et al.  Micro-MR imaging-based computational biomechanics demonstrates reduction in cortical and trabecular bone strength after renal transplantation. , 2012, Radiology.

[77]  Nassir Navab,et al.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2013 , 2013, Lecture Notes in Computer Science.

[78]  H. Song,et al.  Cancellous bone volume and structure in the forearm: noninvasive assessment with MR microimaging and image processing. , 1998, Radiology.

[79]  P Rüegsegger,et al.  Non-invasive bone biopsy: a new method to analyse and display the three-dimensional structure of trabecular bone. , 1994, Physics in medicine and biology.

[80]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[81]  David W. Bates,et al.  CLINICAL USE OF BONE DENSITOMETRY , 1991 .

[82]  Hao Chen,et al.  Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks , 2016, IEEE Transactions on Medical Imaging.

[83]  Ravinder R Regatte,et al.  Finite element analysis applied to 3-T MR imaging of proximal femur microarchitecture: lower bone strength in patients with fragility fractures compared with control subjects. , 2014, Radiology.

[84]  Klaus H. Maier-Hein,et al.  Deep MRI brain extraction: A 3D convolutional neural network for skull stripping , 2016, NeuroImage.

[85]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[86]  Felix Lau,et al.  FastVentricle: Cardiac Segmentation with ENet , 2017, FIMH.

[87]  Ravinder R Regatte,et al.  Feasibility of three‐dimensional MRI of proximal femur microarchitecture at 3 tesla using 26 receive elements without and with parallel imaging , 2014, Journal of magnetic resonance imaging : JMRI.

[88]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[89]  Seunghoon Hong,et al.  Learning Deconvolution Network for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[90]  Thomas Baum,et al.  Characterizing trabecular bone structure for assessing vertebral fracture risk on volumetric quantitative computed tomography , 2015, Medical Imaging.

[91]  Nadia Magnenat-Thalmann,et al.  Robust statistical shape models for MRI bone segmentation in presence of small field of view , 2011, Medical Image Anal..

[92]  R A Zoroofi,et al.  Segmentation of avascular necrosis of the femoral head using 3-D MR images. , 2001, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[93]  S. Goldstein,et al.  Femoral strength is better predicted by finite element models than QCT and DXA. , 1999, Journal of biomechanics.

[94]  Guoyan Zheng,et al.  3 D multi-scale FCN with random modality voxel dropout learning for Intervertebral Disc Localization and Segmentation from Multimodality MR Images , 2018 .

[95]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.