A Deep Machine Learning Method for Concurrent and Interleaved Human Activity Recognition

Human activity recognition has become an important research topic within the field of pervasive computing, ambient assistive living (AAL), robotics, health-care monitoring, and many more. Techniques for recognizing simple and single activities are typical for now, but recognizing complex activities such as concurrent and interleaving activity is still a major challenging issue. In this paper, we propose a two-phase hybrid deep machine learning approach using bi-directional Long-Short Term Memory (BiLSTM) and Skip-Chain Conditional random field (SCCRF) to recognize the complex activity. BiLSTM is a sequential generative deep learning inherited from Recurrent Neural Network (RNN). SCCRFs is a distinctive feature of conditional random field (CRF) that can represent long term dependencies. In the first phase of the proposed approach, we recognized the concurrent activities using the BiLSTM technique, and in the second phase, SCCRF identifies the interleaved activity. Accuracy of the proposed framework against the counterpart state-of-art methods using the publicly available datasets in a smart home environment is analyzed. Our experiment’s result surpasses the previously proposed approaches with an average accuracy of more than 93%.

[1]  Kent Larson,et al.  Activity Recognition in the Home Using Simple and Ubiquitous Sensors , 2004, Pervasive.

[2]  M. Tahar Kechadi,et al.  Human Activity Recognition with Convolutional Neural Networks , 2018, ECML/PKDD.

[3]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[4]  María de Lourdes Martínez-Villaseñor,et al.  A Novel Wearable Sensor-Based Human Activity Recognition Approach Using Artificial Hydrocarbon Networks , 2016, Sensors.

[5]  Michel Galley,et al.  A Skip-Chain Conditional Random Field for Ranking Meeting Utterances by Importance , 2006, EMNLP.

[6]  Changki Lee,et al.  LSTM-CRF Models for Named Entity Recognition , 2017, IEICE Trans. Inf. Syst..

[7]  S. Yoo,et al.  Physical Activity Recognition Using a Single Tri-Axis Accelerometer , 2009 .

[8]  M. Humayun Kabir,et al.  PARE: Profile-Applied Reasoning Engine for Context-Aware System , 2016, Int. J. Distributed Sens. Networks.

[9]  Zesong Fei,et al.  3D-CNN-Based Fused Feature Maps with LSTM Applied to Action Recognition , 2019, Future Internet.

[10]  Wei Xu,et al.  Bidirectional LSTM-CRF Models for Sequence Tagging , 2015, ArXiv.

[11]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[12]  Lawrence B. Holder,et al.  Conditional random fields for activity recognition in smart environments , 2010, IHI.

[13]  Jacques Demongeot,et al.  Health "Smart" home: information technology for patients at home. , 2002, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[14]  Hulya Yalcin Human activity recognition using deep belief networks , 2016, 2016 24th Signal Processing and Communication Application Conference (SIU).

[15]  Matthieu Geist,et al.  Human Activity Recognition Using Recurrent Neural Networks , 2017, CD-MAKE.

[16]  Andrew McCallum,et al.  An Introduction to Conditional Random Fields for Relational Learning , 2007 .

[17]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[18]  Rohan Vaidya,et al.  Handwritten Character Recognition Using Deep-Learning , 2018, 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT).

[19]  Xiaodong Cui,et al.  English Conversational Telephone Speech Recognition by Humans and Machines , 2017, INTERSPEECH.

[20]  Armand Joulin,et al.  Deep Fragment Embeddings for Bidirectional Image Sentence Mapping , 2014, NIPS.

[21]  Andrew McCallum,et al.  Piecewise pseudolikelihood for efficient training of conditional random fields , 2007, ICML '07.

[22]  Gwenn Englebienne,et al.  An activity monitoring system for elderly care using generative and discriminative models , 2010, Personal and Ubiquitous Computing.

[23]  Li Li,et al.  Deep Bi-directional Long Short-Term Memory Model for Short-Term Traffic Flow Prediction , 2017, ICONIP.

[24]  Keshav Thapa,et al.  Log-Viterbi algorithm applied on second-order hidden Markov model for human activity recognition , 2018, Int. J. Distributed Sens. Networks.

[25]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Daniel P. Siewiorek,et al.  Activity recognition and monitoring using multiple sensors on different body positions , 2006, International Workshop on Wearable and Implantable Body Sensor Networks (BSN'06).

[28]  Diane J. Cook,et al.  Keeping the Resident in the Loop: Adapting the Smart Home to the User , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[29]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[30]  Henry A. Kautz,et al.  Inferring High-Level Behavior from Low-Level Sensors , 2003, UbiComp.

[31]  Manuela M. Veloso,et al.  Conditional random fields for activity recognition , 2007, AAMAS '07.

[32]  Ricardo Costa,et al.  Ambient Assisted Living , 2009 .

[33]  Navdeep Jaitly,et al.  Hybrid speech recognition with Deep Bidirectional LSTM , 2013, 2013 IEEE Workshop on Automatic Speech Recognition and Understanding.

[34]  Diane J. Cook,et al.  Human Activity Recognition and Pattern Discovery , 2010, IEEE Pervasive Computing.

[35]  Sung-Hyun Yang,et al.  State-Space based Linear Modeling for Human Activity Recognition in Smart Space , 2018 .

[36]  M. Humayun Kabir,et al.  Two-Layer Hidden Markov Model for Human Activity Recognition in Home Environments , 2016, Int. J. Distributed Sens. Networks.

[37]  Qiang Yang,et al.  CIGAR: Concurrent and Interleaving Goal and Activity Recognition , 2008 .