A framework for automatic hand range of motion evaluation of rheumatoid arthritis patients

Abstract We propose a framework for evaluation of finger movement patterns on Rheumatoid Arthritis patients: flexion, extension, abduction and adduction. The framework uses a state-of-the-art 3D hand pose estimation method that runs in real-time, allowing users to visualize 3D skeleton tracking results at the same time as the depth images are acquired. We compute flexion and abduction angles from the obtained skeleton pose parameters. We performed data acquisition from a cohort of patients and a control set and compared the angles from those two sets of people 1 . An analysis using time series similarity with frequency domain descriptors is adopted to characterize the movement patterns for flexion/extension. We performed classification experiments using these descriptors, thus distinguishing movement sequences of hands with rheumatoid arthritis from healthy hands. The descriptors used in the classification experiment were effective and reached average results of 89 % in scenarios of unseen subjects, and an average of 82 % in experiments with sample synthesis that allow a more robust statistical performance evaluation. Our framework allows the characterization of the current state of the disorder in each patient, with minimal intervention and reduced evaluation time.

[1]  Kyoung Mu Lee,et al.  V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  Frederick Wolfe,et al.  Rheumatoid arthritis , 2010, The Lancet.

[3]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  B Ellis,et al.  Joint angle measurement: a comparative study of the reliability of goniometry and wire tracing for the hand , 1997, Clinical rehabilitation.

[5]  Guijin Wang,et al.  Pose Guided Structured Region Ensemble Network for Cascaded Hand Pose Estimation , 2017, Neurocomputing.

[6]  Vincent Lepetit,et al.  DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[7]  Yang Xiao,et al.  A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation From a Single Depth Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[8]  Zhangyang Wang,et al.  MM-Hand: 3D-Aware Multi-Modal Guided Hand Generation for 3D Hand Pose Synthesis , 2020, ACM Multimedia.

[9]  Qi Ye,et al.  BigHand2.2M Benchmark: Hand Pose Dataset and State of the Art Analysis , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Joy C. MacDermid,et al.  Reliability and Validity of Electro-Goniometric Range of Motion Measurements in Patients with Hand and Wrist Limitations , 2016, The open orthopaedics journal.

[11]  Dinggang Shen,et al.  Editorial: Predictive Intelligence in Biomedical and Health Informatics , 2020, IEEE J. Biomed. Health Informatics.

[12]  Roberto Marcondes Cesar Junior,et al.  Inference from Clustering with Application to Gene-Expression Microarrays , 2002, J. Comput. Biol..

[13]  Luc Van Gool,et al.  Dense 3D Regression for Hand Pose Estimation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Biyao Shao,et al.  3D Room Layout Estimation From a Single RGB Image , 2020, IEEE Transactions on Multimedia.

[15]  Roberto Marcondes Cesar Junior,et al.  Hand range of motion evaluation for Rheumatoid Arthritis patients , 2019, FG.

[16]  Van-Hung Le,et al.  A Survey on 3D Hand Skeleton and Pose Estimation by Convolutional Neural Network , 2020 .

[17]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  G. Passos,et al.  Cigarette smoke induces miR-132 in Th17 cells that enhance osteoclastogenesis in inflammatory arthritis , 2020, Proceedings of the National Academy of Sciences.

[19]  Daniel Thalmann,et al.  3D Convolutional Neural Networks for Efficient and Robust Hand Pose Estimation from Single Depth Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Hong Yan,et al.  A Machine Learning Approach to Improve Contactless Heart Rate Monitoring Using a Webcam , 2014, IEEE Journal of Biomedical and Health Informatics.

[21]  Björn Stenger,et al.  Model-based hand tracking using a hierarchical Bayesian filter , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Anne Bruton,et al.  Comparison of Visual Estimation and Goniometry for Assessment of Metacarpophalangeal Joint Angle , 1999 .

[23]  Carlos Alberto Fortulan,et al.  A new concept of orthosis for correcting fingers ulnar deviation , 2017 .

[24]  B. Jolles,et al.  Reliability of the Knee Smartphone-Application Goniometer in the Acute Orthopedic Setting , 2016, The Journal of Knee Surgery.

[25]  Fei Qiao,et al.  Region ensemble network: Improving convolutional network for hand pose estimation , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[26]  Paulo Louzada-Junior,et al.  Guidelines for the diagnosis of rheumatoid arthritis. , 2013, Revista brasileira de reumatologia.

[27]  Vincent Lepetit,et al.  Hands Deep in Deep Learning for Hand Pose Estimation , 2015, ArXiv.

[28]  Antonis A. Argyros,et al.  Back to RGB: 3D Tracking of Hands and Hand-Object Interactions Based on Short-Baseline Stereo , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[29]  Antonis A. Argyros,et al.  Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[30]  Thomas Brox,et al.  Learning to Estimate 3D Hand Pose from Single RGB Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  S. Desale,et al.  Viability of Hand and Wrist Photogoniometry , 2018, Hand.

[32]  Yue Gao,et al.  Deep Multi-View Enhancement Hashing for Image Retrieval , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Sergio Escalera,et al.  Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[34]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[35]  Luc Van Gool,et al.  Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Zhanpeng Jin,et al.  Enabling Smart Personalized Healthcare: A Hybrid Mobile-Cloud Approach for ECG Telemonitoring , 2014, IEEE Journal of Biomedical and Health Informatics.

[37]  Christian Theobalt,et al.  GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Li Liu,et al.  JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image , 2020, ECCV.

[39]  Pavlo Molchanov,et al.  Hand gesture recognition with 3D convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[40]  Yongdong Zhang,et al.  Depth Image Denoising Using Nuclear Norm and Learning Graph Model , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[41]  Yimin Zhou,et al.  A novel finger and hand pose estimation technique for real-time hand gesture recognition , 2016, Pattern Recognit..