Closing the Wearable Gap: Foot–ankle kinematic modeling via deep learning models based on a smart sock wearable

Abstract The development of wearable technology, which enables motion tracking analysis for human movement outside the laboratory, can improve awareness of personal health and performance. This study used a wearable smart sock prototype to track foot–ankle kinematics during gait movement. Multivariable linear regression and two deep learning models, including long short-term memory (LSTM) and convolutional neural networks, were trained to estimate the joint angles in sagittal and frontal planes measured by an optical motion capture system. Participant-specific models were established for ten healthy subjects walking on a treadmill. The prototype was tested at various walking speeds to assess its ability to track movements for multiple speeds and generalize models for estimating joint angles in sagittal and frontal planes. LSTM outperformed other models with lower mean absolute error (MAE), lower root mean squared error, and higher R-squared values. The average MAE score was less than 1.138° and 0.939° in sagittal and frontal planes, respectively, when training models for each speed and 2.15° and 1.14° when trained and evaluated for all speeds. These results indicate wearable smart socks to generalize foot–ankle kinematics over various walking speeds with relatively low error and could consequently be used to measure gait parameters without the need for a lab-constricted motion capture system.

[1]  Hong Wang,et al.  Continuous and simultaneous estimation of lower limb multi-joint angles from sEMG signals based on stacked convolutional and LSTM models , 2022, Expert Syst. Appl..

[2]  Alireza Bab-Hadiashar,et al.  Robust Pooling through the Data Mode , 2021, Intell. Syst. Appl..

[3]  Sachini N. K. Kodithuwakku Arachchige,et al.  Closing the Wearable Gap—Part VIII: A Validation Study for a Smart Knee Brace to Capture Knee Joint Kinematics , 2021 .

[4]  D. Gouwanda,et al.  Joint angle estimation with wavelet neural networks , 2021, Scientific Reports.

[5]  Sanam Moghaddamnia,et al.  Efficiency of deep neural networks for joint angle modeling in digital gait assessment , 2021, EURASIP Journal on Advances in Signal Processing.

[6]  Brian K. Smith,et al.  Closing the Wearable Gap-Part VII: A Retrospective of Stretch Sensor Tool Kit Development for Benchmark Testing , 2020, Electronics.

[7]  Bernd Markert,et al.  Artificial Neural Networks in Motion Analysis—Applications of Unsupervised and Heuristic Feature Selection Techniques , 2020, Sensors.

[8]  Brian K. Smith,et al.  Closing the Wearable Gap—Part VI: Human Gait Recognition Using Deep Learning Methodologies , 2020, Electronics.

[9]  Reuben F. Burch,et al.  State-of-the-art review of athletic wearable technology: What 113 strength and conditioning coaches and athletic trainers from the USA said about technology in sports , 2020, International Journal of Sports Science & Coaching.

[10]  Brian K. Smith,et al.  Closing the Wearable Gap—Part V: Development of a Pressure-Sensitive Sock Utilizing Soft Sensors , 2019, Sensors.

[11]  Carlo Menon,et al.  Lower Body Kinematics Monitoring in Running Using Fabric-Based Wearable Sensors and Deep Convolutional Neural Networks , 2019, Sensors.

[12]  Brian K. Smith,et al.  Closing the Wearable Gap—Part IV: 3D Motion Capture Cameras Versus Soft Robotic Sensors Comparison of Gait Movement Assessment , 2019, Electronics.

[13]  Brian Smith,et al.  Closing the Wearable Gap—Part III: Use of Stretch Sensors in Detecting Ankle Joint Kinematics During Unexpected and Expected Slip and Trip Perturbations , 2019, Electronics.

[14]  Brian K. Smith,et al.  Closing the Wearable Gap—Part II: Sensor Orientation and Placement for Foot and Ankle Joint Kinematic Measurements , 2019, Sensors.

[15]  Alex Sherstinsky,et al.  Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network , 2018, Physica D: Nonlinear Phenomena.

[16]  Maury A. Nussbaum,et al.  Preferred Placement and Usability of a Smart Textile System vs. Inertial Measurement Units for Activity Monitoring , 2018, Sensors.

[17]  James E Ball,et al.  Closing the Wearable Gap: Mobile Systems for Kinematic Signal Monitoring of the Foot and Ankle , 2018, Electronics.

[18]  Tao Mei,et al.  Wearable Stretch Sensors for Motion Measurement of the Wrist Joint Based on Dielectric Elastomers , 2017, Sensors.

[19]  Jim Tørresen,et al.  A robust gait recognition system using spatiotemporal features and deep learning , 2017, 2017 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI).

[20]  John C. Garner,et al.  Slip initiation in alternative and slip-resistant footwear , 2017, International journal of occupational safety and ergonomics : JOSE.

[21]  Ahmet Turan Özdemir,et al.  An Analysis on Sensor Locations of the Human Body for Wearable Fall Detection Devices: Principles and Practice , 2016, Sensors.

[22]  Namita Mittal,et al.  A framework for human recognition using a multimodel Gait analysis approach , 2016, 2016 International Conference on Computing, Communication and Automation (ICCCA).

[23]  Kimberly D. Kendricks,et al.  Deep network for analyzing gait patterns in low resolution video towards threat identification. , 2016 .

[24]  John C. Garner,et al.  Heel contact dynamics in alternative footwear during slip events , 2015 .

[25]  Robert J. Wood,et al.  Wearable soft sensing suit for human gait measurement , 2014, Int. J. Robotics Res..

[26]  Matthew Rhudy,et al.  Time Alignment Techniques for Experimental Sensor Data , 2014 .

[27]  Hermie Hermens,et al.  Optimal Sensor Placement for Measuring Physical Activity with a 3D Accelerometer , 2014, Sensors.

[28]  Philippe C Dixon,et al.  Ankle and midfoot kinetics during normal gait: a multi-segment approach. , 2012, Journal of biomechanics.

[29]  Tilak Dias,et al.  Automatic identification of gait events using an instrumented sock , 2011, Journal of NeuroEngineering and Rehabilitation.

[30]  Yann LeCun,et al.  Convolutional networks and applications in vision , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[31]  P. Beek,et al.  Is slow walking more stable? , 2009, Journal of biomechanics.

[32]  Siegfried Bauer,et al.  Capacitive extensometry for transient strain analysis of dielectric elastomer actuators , 2008 .

[33]  M. Hanlon,et al.  Prediction methods to account for the effect of gait speed on lower limb angular kinematics. , 2006, Gait & posture.

[34]  E. Delahunt,et al.  Ankle function during gait in patients with chronic ankle instability compared to controls. , 2006, Clinical biomechanics.

[35]  B. MacWilliams,et al.  Foot kinematics and kinetics during adolescent gait. , 2003, Gait & posture.

[36]  W. Eisma,et al.  Walking speed of normal subjects and amputees: Aspects of validity of gait analysis , 1993, Prosthetics and orthotics international.

[37]  R. Shephard PAR-Q, Canadian Home Fitness Test and Exercise Screening Alternatives , 1988, Sports medicine.

[38]  Reuben F. Burch,et al.  Closing the Wearable Gap–Part IX: Validation of an Improved Ankle Motion Capture Wearable , 2021, IEEE Access.

[39]  Hefei Ling,et al.  Multi-View Gait Recognition Based on a Spatial-Temporal Deep Neural Network , 2018, IEEE Access.

[40]  S. M. N. Arosha Senanayake,et al.  Neural computing for walking gait pattern identification based on multi-sensor data fusion of lower limb muscles , 2016, Neural Computing and Applications.

[41]  Leendert Schaake,et al.  Optimal Sensor Placement for Measuring Physical Activity with a 3 D Accelerometer , 2014 .

[42]  Ulrich Rückert,et al.  Athlete Identification using Acceleration and Electrocardiographic Measurements Recorded with a Wireless Body Sensor , 2013, BIOSIGNALS.

[43]  A. Nicol,et al.  A multi-segment kinematic model of the foot with a novel definition of forefoot motion for use in clinical gait analysis during walking. , 2007, Journal of biomechanics.

[44]  Jean-Yves Audibert Optimization for Machine Learning , 1995 .

[45]  Yves Chauvin,et al.  Backpropagation: theory, architectures, and applications , 1995 .

[46]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.