KNEEL: Knee Anatomical Landmark Localization Using Hourglass Networks

This paper addresses the challenge of localization of anatomical landmarks in knee X-ray images at different stages of osteoarthritis (OA). Landmark localization can be viewed as regression problem, where the landmark position is directly predicted by using the region of interest or even full-size images leading to large memory footprint, especially in case of high resolution medical images. In this work, we propose an efficient deep neural networks framework with an hourglass architecture utilizing a soft-argmax layer to directly predict normalized coordinates of the landmark points. We provide an extensive evaluation of different regularization techniques and various loss functions to understand their influence on the localization performance. Furthermore, we introduce the concept of transfer learning from low-budget annotations, and experimentally demonstrate that such approach is improving the accuracy of landmark localization. Compared to the prior methods, we validate our model on two datasets that are independent from the train data and assess the performance of the method for different stages of OA severity. The proposed approach demonstrates better generalization performance compared to the current state-of-the-art.

[1]  J. Kellgren,et al.  Radiological Assessment of Osteo-Arthrosis , 1957, Annals of the rheumatic diseases.

[2]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  C. Peterfy,et al.  Fixed-flexion radiography of the knee provides reproducible joint space width measurements in osteoarthritis , 2004, European Radiology.

[5]  Timothy F. Cootes,et al.  Feature Detection and Tracking with Constrained Local Models , 2006, BMVC.

[6]  Mingrui Wu,et al.  Gradient descent optimization of smoothed information retrieval metrics , 2010, Information Retrieval.

[7]  Stefanos Zafeiriou,et al.  Subspace Learning from Image Gradient Orientations , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Timothy F. Cootes,et al.  Accurate Bone Segmentation in 2D Radiographs Using Fully Automatic Shape Model Matching Based On Regression-Voting , 2013, MICCAI.

[9]  Stefanos Zafeiriou,et al.  Menpo: A Comprehensive Platform for Parametric Image Alignment and Visual Deformable Models , 2014, ACM Multimedia.

[10]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[11]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[12]  Timothy F. Cootes,et al.  Automated Shape and Texture Analysis for Detection of Osteoarthritis from Radiographs of the Knee , 2015, MICCAI.

[13]  Claudia Lindner,et al.  Robust and Accurate Shape Model Matching Using Random Forest Regression-Voting. , 2015, IEEE transactions on pattern analysis and machine intelligence.

[14]  K. Allen,et al.  Epidemiology of osteoarthritis: state of the evidence , 2015 .

[15]  Rachid Jennane,et al.  ROI impact on the characterization of knee osteoarthritis using fractal analysis , 2015, 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA).

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  J. Niinimäki,et al.  Comparison of Diagnostic Performance of Semi-Quantitative Knee Ultrasound and Knee Radiography with MRI: Oulu Knee Osteoarthritis Study , 2016, Scientific Reports.

[18]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[19]  Georgios Tzimiropoulos,et al.  Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resources , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[20]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[21]  Simo Saarakkala,et al.  A Novel Method for Automatic Localization of Joint Area on Knee Plain Radiographs , 2017, SCIA.

[22]  Noel E. O'Connor,et al.  Automatic Detection of Knee Joints and Quantification of Knee Osteoarthritis Severity Using Convolutional Neural Networks , 2017, MLDM.

[23]  Graham W. Taylor,et al.  Improved Regularization of Convolutional Neural Networks with Cutout , 2017, ArXiv.

[24]  Hongyi Zhang,et al.  mixup: Beyond Empirical Risk Minimization , 2017, ICLR.

[25]  Qiang Ji,et al.  Facial Landmark Detection: A Literature Survey , 2018, International Journal of Computer Vision.

[26]  Thomas M. Link,et al.  Applying Densely Connected Convolutional Neural Networks for Staging Osteoarthritis Severity from Plain Radiographs , 2018, Journal of Digital Imaging.

[27]  Josef Kittler,et al.  Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Simo Saarakkala,et al.  Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs: A Deep Learning-Based Approach , 2017, Scientific Reports.

[29]  Sina Honari,et al.  Improving Landmark Localization with Semi-Supervised Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Abhishek Dutta,et al.  The VIA Annotation Software for Images, Audio and Video , 2019, ACM Multimedia.

[31]  Christian Payer,et al.  Integrating spatial configuration into heatmap regression based CNNs for landmark localization , 2019, Medical Image Anal..

[32]  Simo Saarakkala,et al.  Automatic Grading of Individual Knee Osteoarthritis Features in Plain Radiographs Using Deep Convolutional Neural Networks , 2019, Diagnostics.

[33]  Xiaoshuang Shi,et al.  Fully automatic knee osteoarthritis severity grading using deep neural networks with a novel ordinal loss , 2019, Comput. Medical Imaging Graph..

[34]  Josef Kittler,et al.  Mining Hard Augmented Samples for Robust Facial Landmark Localization With CNNs , 2019, IEEE Signal Processing Letters.

[35]  Jonathan T. Barron,et al.  A General and Adaptive Robust Loss Function , 2017, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Torsten Sattler,et al.  DGC-Net: Dense Geometric Correspondence Network , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[37]  E. Lespessailles,et al.  A decision support tool for early detection of knee OsteoArthritis using X-ray imaging and machine learning: Data from the OsteoArthritis Initiative , 2019, Comput. Medical Imaging Graph..

[38]  Juho Kannala,et al.  Geometric Image Correspondence Verification by Dense Pixel Matching , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[39]  P. Alam ‘G’ , 2021, Composites Engineering: An A–Z Guide.

[40]  P. Alam ‘N’ , 2021, Composites Engineering: An A–Z Guide.