Two-dimensional video-based analysis of human gait using pose estimation

Walking is the primary mode of human locomotion. Accordingly, people have been interested in studying human gait since at least the fourth century BC. Human gait analysis is now common in many fields of clinical and basic research, but gold standard approaches – e.g., three-dimensional motion capture, instrumented mats or footwear, and wearables – are often expensive, immobile, data-limited, and/or require specialized equipment or expertise for operation. Recent advances in video-based pose estimation have suggested exciting potential for analyzing human gait using only two-dimensional video inputs collected from readily accessible devices (e.g., smartphones, tablets). However, we currently lack: 1) data about the accuracy of video-based pose estimation approaches for human gait analysis relative to gold standard measurement techniques and 2) an available workflow for performing human gait analysis via video-based pose estimation. In this study, we compared a large set of spatiotemporal and sagittal kinematic gait parameters as measured by OpenPose (a freely available algorithm for video-based human pose estimation) and three-dimensional motion capture from trials where healthy adults walked overground. We found that OpenPose performed well in estimating many gait parameters (e.g., step time, step length, sagittal hip and knee angles) while some (e.g., double support time, sagittal ankle angles) were less accurate. We observed that mean values for individual participants – as are often of primary interest in clinical settings – were more accurate than individual step-by-step measurements. We also provide a workflow for users to perform their own gait analyses and offer suggestions and considerations for future approaches.

[1]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[2]  Cewu Lu,et al.  RMPE: Regional Multi-person Pose Estimation , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Tieniu Tan,et al.  Silhouette Analysis-Based Gait Recognition for Human Identification , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  C. Cobelli,et al.  A Markerless Motion Capture System to Study Musculoskeletal Biomechanics: Visual Hull and Simulated Annealing Approach , 2006, Annals of Biomedical Engineering.

[5]  J S Higginson,et al.  Two simple methods for determining gait events during treadmill and overground walking using kinematic data. , 2008, Gait & posture.

[6]  Xu Xu,et al.  Accuracy of the Microsoft Kinect for measuring gait parameters during treadmill walking. , 2015, Gait & posture.

[7]  Atsushi Iwata,et al.  Quantifying normal and parkinsonian gait features from home movies: Practical application of a deep learning–based 2D pose estimator , 2019, bioRxiv.

[8]  James Nga-Kwok Liu,et al.  Gait flow image: A silhouette-based gait representation for human identification , 2011, Pattern Recognit..

[9]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Peter V. Gehler,et al.  DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Dimitris Kastaniotis,et al.  A framework for gait-based recognition using Kinect , 2015, Pattern Recognit. Lett..

[12]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Ross A Clark,et al.  Gait assessment using the Microsoft Xbox One Kinect: Concurrent validity and inter-day reliability of spatiotemporal and kinematic variables. , 2015, Journal of biomechanics.

[14]  Matthias Bethge,et al.  Using DeepLabCut for 3D markerless pose estimation across species and behaviors , 2018 .

[15]  Simon Winkelbach,et al.  Hierarchical pose estimation for human gait analysis , 2012, Comput. Methods Programs Biomed..

[16]  Melvyn Roerdink,et al.  Kinematic Validation of a Multi-Kinect v2 Instrumented 10-Meter Walkway for Quantitative Gait Assessments , 2015, PloS one.

[17]  Robert T. Collins,et al.  Silhouette-based human identification from body shape and gait , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[18]  Bernt Schiele,et al.  DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model , 2016, ECCV.

[19]  Richard Baker,et al.  The history of gait analysis before the advent of modern computers. , 2007, Gait & posture.

[20]  Marco Tarabini,et al.  3D Tracking of Human Motion Using Visual Skeletonization and Stereoscopic Vision , 2020, Frontiers in Bioengineering and Biotechnology.

[21]  Hans-Peter Seidel,et al.  EgoCap , 2016, ACM Trans. Graph..

[22]  Alexandra Pfister,et al.  Comparative abilities of Microsoft Kinect and Vicon 3D motion capture for gait analysis , 2014, Journal of medical engineering & technology.

[23]  Ruigang Yang,et al.  Accuracy and repeatability of joint angles measured using a single camera markerless motion capture system. , 2014, Journal of biomechanics.

[24]  N K LiuJames,et al.  Gait flow image , 2011 .

[25]  James J. Little,et al.  A Simple Yet Effective Baseline for 3d Human Pose Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Tathagata Ray,et al.  Human Gait Analysis Using OpenPose , 2019, 2019 Fifth International Conference on Image Information Processing (ICIIP).

[27]  Bernt Schiele,et al.  2D Human Pose Estimation: New Benchmark and State of the Art Analysis , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Jing Li,et al.  View-invariant gait recognition based on kinect skeleton feature , 2018, Multimedia Tools and Applications.

[29]  TheodorakopoulosIlias,et al.  A framework for gait-based recognition using Kinect , 2015 .

[30]  Bogdan Kwolek,et al.  Calibrated and synchronized multi-view video and motion capture dataset for evaluation of gait recognition , 2019, Multimedia Tools and Applications.

[31]  Thomas P Andriacchi,et al.  The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications , 2006, Journal of NeuroEngineering and Rehabilitation.

[32]  Guodong Guo,et al.  Classification of Normal and Pathological Gait in Young Children Based on Foot Pressure Data , 2016, Neuroinformatics.

[33]  Cong Wang,et al.  Silhouette-based gait recognition via deterministic learning , 2014, Pattern Recognit..

[34]  Megumi Ota,et al.  Verification of reliability and validity of motion analysis systems during bilateral squat using human pose tracking algorithm. , 2020, Gait & posture.

[35]  Konrad Kording,et al.  Pose estimates from online videos show that side-by-side walkers synchronize movement under naturalistic conditions , 2019, PloS one.

[36]  Ross A Clark,et al.  Three-dimensional cameras and skeleton pose tracking for physical function assessment: A review of uses, validity, current developments and Kinect alternatives. , 2019, Gait & posture.

[37]  Kevin M. Cury,et al.  DeepLabCut: markerless pose estimation of user-defined body parts with deep learning , 2018, Nature Neuroscience.

[38]  Yoichi Iino,et al.  Evaluation of 3D Markerless Motion Capture Accuracy Using OpenPose With Multiple Video Cameras , 2020, Frontiers in Sports and Active Living.