From Human Pose to On-Body Devices for Human-Activity Recognition

Human Activity Recognition (HAR), using inertial measurements from on-body devices, has not seen a great advantage from deep architectures. This drawback is mainly due to the lack of annotated data, diversity of on-body device configurations, the class-unbalance problem, and non-standard human activity definitions. Approaches for improving the performance of such architectures, e.g., transfer learning, are therefore difficult to apply. This paper introduces a method for transfer learning from human-pose estimations as a source for improving HAR using inertial measurements obtained from on-body devices. We propose to fine-tune deep architectures, trained using sequences of human poses from a large dataset and their derivatives, for solving HAR on inertial measurements from on-body devices. Derivatives of human poses will be considered as a sort of synthetic data for HAR. We deploy two different temporal-convolutional architectures as classifiers. An evaluation of the method is carried out on three benchmark datasets improving the classification performance.

[1]  Michael ten Hompel,et al.  Human machine synergies in intra-logistics: Creating a hybrid network for research and technologies , 2017, 2017 Federated Conference on Computer Science and Information Systems (FedCSIS).

[2]  Gwenn Englebienne,et al.  In-Home Activity Recognition: Bayesian Inference for Hidden Markov Models , 2014, IEEE Pervasive Computing.

[3]  Duc A. Tran,et al.  The 11th International Conference on Mobile Systems and Pervasive Computing (MobiSPC-2014) A Study on Human Activity Recognition Using Accelerometer Data from Smartphones , 2014 .

[4]  Gernot A. Fink,et al.  Deep Neural Network based Human Activity Recognition for the Order Picking Process , 2017, iWOAR.

[5]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[6]  David Picard,et al.  2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[8]  Gernot A. Fink,et al.  Towards a Framework for Semi-Automated Annotation of Human Order Picking Activities Using Motion Capturing , 2018, 2018 Federated Conference on Computer Science and Information Systems (FedCSIS).

[9]  Gernot A. Fink,et al.  Convolutional Neural Networks for Human Activity Recognition Using Body-Worn Sensors , 2018, Informatics.

[10]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[11]  Sebastian Sudholt,et al.  Learning Deep Representations for Word Spotting under Weak Supervision , 2017, 2018 13th IAPR International Workshop on Document Analysis Systems (DAS).

[12]  Damith Chinthana Ranasinghe,et al.  Efficient dense labelling of human activity sequences from wearables using fully convolutional networks , 2018, Pattern Recognit..

[13]  Rainer Stiefelhagen,et al.  CNN-based sensor fusion techniques for multimodal human activity recognition , 2017, SEMWEB.

[14]  Niall Twomey,et al.  A Comprehensive Study of Activity Recognition Using Accelerometers , 2018, Informatics.

[15]  Andrew Zisserman,et al.  Convolutional Two-Stream Network Fusion for Video Action Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Jie Yu,et al.  Multi-Modal Three-Stream Network for Action Recognition , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[17]  Kristof Van Laerhoven,et al.  Using Wrist-Worn Activity Recognition for Basketball Game Analysis , 2018, iWOAR.

[18]  Gernot A. Fink,et al.  Human Activity Recognition for Production and Logistics - A Systematic Literature Review , 2019, Inf..

[19]  Ricardo Chavarriaga,et al.  The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition , 2013, Pattern Recognit. Lett..

[20]  Belkacem Chikhaoui,et al.  A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors , 2018, MLDM.

[21]  Sung-Bae Cho,et al.  Deep Convolutional Neural Networks for Human Activity Recognition with Smartphone Sensors , 2015, ICONIP.

[22]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[23]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[25]  Sung-Bae Cho,et al.  Human activity recognition with smartphone sensors using deep learning neural networks , 2016, Expert Syst. Appl..

[26]  Yizhou Yu,et al.  Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-Tuning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Marian Haescher,et al.  Mobile Assisted Living: Smartwatch-based Fall Risk Assessment for Elderly People , 2018, iWOAR.

[28]  Nirmalya Roy,et al.  Unseen Activity Recognitions: A Hierarchical Active Transfer Learning Approach , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[29]  Gernot A. Fink,et al.  Motion Classification for Analyzing the Order Picking Process using Mobile Sensors - General Concepts, Case Studies and Empirical Evaluation , 2016, ICPRAM.

[30]  Daniel Roggen,et al.  Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations , 2016, SEMWEB.

[31]  Didier Stricker,et al.  Introducing a New Benchmarked Dataset for Activity Monitoring , 2012, 2012 16th International Symposium on Wearable Computers.

[32]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[33]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.