IMU-to-Segment Assignment and Orientation Alignment for the Lower Body Using Deep Learning

Human body motion analysis based on wearable inertial measurement units (IMUs) receives a lot of attention from both the research community and the and industrial community. This is due to the significant role in, for instance, mobile health systems, sports and human computer interaction. In sensor based activity recognition, one of the major issues for obtaining reliable results is the sensor placement/assignment on the body. For inertial motion capture (joint kinematics estimation) and analysis, the IMU-to-segment (I2S) assignment and alignment are central issues to obtain biomechanical joint angles. Existing approaches for I2S assignment usually rely on hand crafted features and shallow classification approaches (e.g., support vector machines), with no agreement regarding the most suitable features for the assignment task. Moreover, estimating the complete orientation alignment of an IMU relative to the segment it is attached to using a machine learning approach has not been shown in literature so far. This is likely due to the high amount of training data that have to be recorded to suitably represent possible IMU alignment variations. In this work, we propose online approaches for solving the assignment and alignment tasks for an arbitrary amount of IMUs with respect to a biomechanical lower body model using a deep learning architecture and windows of 128 gyroscope and accelerometer data samples. For this, we combine convolutional neural networks (CNNs) for local filter learning with long-short-term memory (LSTM) recurrent networks as well as generalized recurrent units (GRUs) for learning time dynamic features. The assignment task is casted as a classification problem, while the alignment task is casted as a regression problem. In this framework, we demonstrate the feasibility of augmenting a limited amount of real IMU training data with simulated alignment variations and IMU data for improving the recognition/estimation accuracies. With the proposed approaches and final models we achieved 98.57% average accuracy over all segments for the I2S assignment task (100% when excluding left/right switches) and an average median angle error over all segments and axes of 2.91° for the I2S alignment task.

[1]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Patrice Y. Simard,et al.  Best practices for convolutional neural networks applied to visual document analysis , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[4]  Paul Lukowicz,et al.  Where am I: Recognizing On-body Positions of Wearable Sensors , 2005, LoCA.

[5]  Paul Lukowicz,et al.  Using acceleration signatures from everyday activities for on-body device location , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[6]  Paul Lukowicz,et al.  Dealing with sensor displacement in motion-based onbody activity recognition systems , 2008, UbiComp.

[7]  B M Jolles,et al.  Functional calibration procedure for 3D knee joint angle description using inertial sensors. , 2009, Journal of biomechanics.

[8]  大西 明宏.  使いたさ発展中 NaturalPoint 社 OptiTrack , 2009 .

[9]  D. Roetenberg,et al.  Xsens MVN: Full 6DOF Human Motion Tracking Using Miniature Inertial Sensors , 2009 .

[10]  F C T van der Helm,et al.  Functionally interpretable local coordinate systems for the upper extremity using inertial & magnetic measurement systems. , 2010, Journal of biomechanics.

[11]  Surapa Thiemjarus,et al.  A Device-Orientation Independent Method for Activity Recognition , 2010, 2010 International Conference on Body Sensor Networks.

[12]  Daniel Tik-Pui Fong,et al.  The Use of Wearable Inertial Motion Sensors in Human Lower Limb Biomechanics Studies: A Systematic Review , 2010, Sensors.

[13]  Majid Sarrafzadeh,et al.  Accelerometer-based on-body sensor localization for health and medical monitoring applications , 2011, Pervasive Mob. Comput..

[14]  Surapa Thiemjarus,et al.  Accurate Activity Recognition Using a Mobile Phone Regardless of Device Orientation and Location , 2011, 2011 International Conference on Body Sensor Networks.

[15]  Jie Liu,et al.  A rotation based method for detecting on-body positions of mobile devices , 2011, UbiComp '11.

[16]  Shyamal Patel,et al.  A review of wearable sensors and systems with application in rehabilitation , 2012, Journal of NeuroEngineering and Rehabilitation.

[17]  Hongyi Li,et al.  A method to deal with installation errors of wearable accelerometers for human activity recognition , 2011, Physiological measurement.

[18]  D. K. Arvind,et al.  IMUSim: A simulation environment for inertial sensing algorithm design and evaluation , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[19]  Guido Bugmann,et al.  A Recipe on the Parameterization of Rotation Matrices for Non-Linear Optimization using Quaternions , 2012 .

[20]  Majid Sarrafzadeh,et al.  Robust Human Activity and Sensor Location Corecognition via Sparse Signal Representation , 2012, IEEE Transactions on Biomedical Engineering.

[21]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[22]  Peter H Veltink,et al.  Automatic identification of inertial sensor placement on human body segments during walking , 2013, Journal of NeuroEngineering and Rehabilitation.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Yacine Challal,et al.  Wireless sensor networks for rehabilitation applications: Challenges and opportunities , 2013, J. Netw. Comput. Appl..

[26]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[27]  Carmen C. Y. Poon,et al.  Unobtrusive Sensing and Wearable Devices for Health Informatics , 2014, IEEE Transactions on Biomedical Engineering.

[28]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[29]  Eduardo Palermo,et al.  Experimental evaluation of accuracy and repeatability of a novel body-to-sensor calibration procedure for inertial sensor-based gait analysis , 2014 .

[30]  Paul Lukowicz,et al.  Sensor Placement Variations in Wearable Activity Recognition , 2014, IEEE Pervasive Computing.

[31]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[32]  Xi Zhang,et al.  Learning Classifiers from Synthetic Data Using a Multichannel Autoencoder , 2015, ArXiv.

[33]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[34]  Sébastien Changey,et al.  Magnetometer-Augmented IMU Simulator: In-Depth Elaboration , 2015, Sensors.

[35]  Xi Zhang,et al.  Learning from Synthetic Data Using a Stacked Multichannel Autoencoder , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[36]  Angelo M. Sabatini,et al.  Accelerometry-based recognition of the placement sites of a wearable sensor , 2015, Pervasive Mob. Comput..

[37]  Faicel Chamroukhi,et al.  Physical Human Activity Recognition Using Wearable Sensors , 2015, Sensors.

[38]  Brice Bouvier,et al.  Upper Limb Kinematics Using Inertial and Magnetic Sensors: Comparison of Sensor-to-Segment Calibrations , 2015, Sensors.

[39]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[40]  Zoubin Ghahramani,et al.  A Theoretically Grounded Application of Dropout in Recurrent Neural Networks , 2015, NIPS.

[41]  Jun Hu,et al.  Activity recognition based on inertial sensors for Ambient Assisted Living , 2016, 2016 19th International Conference on Information Fusion (FUSION).

[42]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[43]  Bertram Taetz,et al.  Towards self-calibrating inertial body motion capture , 2016, 2016 19th International Conference on Information Fusion (FUSION).

[44]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[45]  Daniel Roggen,et al.  Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations , 2016, SEMWEB.

[46]  Hwee Pink Tan,et al.  Deep Activity Recognition Models with Triaxial Accelerometers , 2015, AAAI Workshop: Artificial Intelligence Applied to Assistive Technologies and Smart Environments.

[47]  Bertram Taetz,et al.  On Inertial Body Tracking in the Presence of Model Calibration Errors , 2016, Sensors.

[48]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[49]  Angelo M. Sabatini,et al.  Assessing the Performance of Sensor Fusion Methods: Application to Magnetic-Inertial-Based Human Body Tracking , 2016, Sensors.

[50]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[51]  Thomas Seel,et al.  Automatic pairing of inertial sensors to lower limb segments – a plug-and-play approach , 2016 .

[52]  Didier Stricker,et al.  Two Phase Classification for Early Hand Gesture Recognition in 3D Top View Data , 2016, ISVC.

[53]  Bertram Taetz,et al.  Development of an Inertial Motion Capture System for Clinical Application , 2017, i-com.

[54]  Didier Stricker,et al.  A framework for an accurate point cloud based registration of full 3D human body scans , 2017, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[55]  Fredrik Olsson,et al.  Experimental evaluation of joint position estimation using inertial sensors , 2017, 2017 20th International Conference on Information Fusion (Fusion).

[56]  Frank D. Wood,et al.  Using synthetic data to train neural networks is model-based reasoning , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[57]  Didier Stricker,et al.  [POSTER] A Probabilistic Combination of CNN and RNN Estimates for Hand Gesture Based Interaction in Car , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[58]  Surapa Thiemjarus,et al.  Analysis of Optimal Sensor Positions for Activity Classification and Application on a Different Data Collection Scenario , 2017, Sensors.

[59]  Ivan Tashev,et al.  Unsupervised deep representation learning to remove motion artifacts in free-mode body sensor networks , 2017, 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[60]  Bertram Taetz,et al.  Real-time inertial lower body kinematics and ground contact estimation at anatomical foot points for agile human locomotion , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).