On the Generality of Codebook Approach for Sensor-Based Human Activity Recognition

With the recent spread of mobile devices equipped with different sensors, it is possible to continuously recognise and monitor activities in daily life. This sensor-based human activity recognition is formulated as sequence classification to categorise sequences of sensor values into appropriate activity classes. One crucial problem is how to model features that can precisely represent characteristics of each sequence and lead to accurate recognition. It is laborious and/or difficult to hand-craft such features based on prior knowledge and manual investigation about sensor data. To overcome this, we focus on a feature learning approach that extracts useful features from a large amount of data. In particular, we adopt a simple but effective one, called codebook approach, which groups numerous subsequences collected from sequences into clusters. Each cluster centre is called a codeword and represents a statistically distinctive subsequence. Then, a sequence is encoded as a feature expressing the distribution of codewords. The extensive experiments on different recognition tasks for physical, mental and eye-based activities validate the effectiveness, generality and usability of the codebook approach.

[1]  Kimiaki Shirahama,et al.  Towards large-scale multimedia retrieval enriched by knowledge about human interpretation , 2014, Multimedia Tools and Applications.

[2]  Stefano Paolucci,et al.  Wearable inertial sensors for human movement analysis , 2016, Expert review of medical devices.

[3]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[4]  M. Grzegorzek,et al.  Towards Large-Scale Multimedia Retrieval Enriched by Knowledge about Human Interpretation Retrospective Survey , 2014 .

[5]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[6]  Carsten Rother,et al.  Weakly supervised discriminative localization and classification: a joint learning process , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[8]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Jian Lu,et al.  A Pattern Mining Approach to Sensor-Based Human Activity Recognition , 2011, IEEE Transactions on Knowledge and Data Engineering.

[10]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[11]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[12]  Michael G. Madden,et al.  An Ensemble Dynamic Time Warping Classifier with Application to Activity Recognition , 2010, SGAI Conf..

[13]  Angelica Munoz-Melendez,et al.  Wearable Inertial Sensors for Human Motion Analysis: A Review , 2016, IEEE Sensors Journal.

[14]  Greg Mori,et al.  Latent Maximum Margin Clustering , 2013, NIPS.

[15]  Jian Pei,et al.  Data Mining: Concepts and Techniques, 3rd edition , 2006 .

[16]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Cor J. Veenman,et al.  Visual Word Ambiguity , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Frédéric Jurie,et al.  Sampling Strategies for Bag-of-Features Image Classification , 2006, ECCV.

[19]  Eamonn J. Keogh,et al.  Clustering Time Series Using Unsupervised-Shapelets , 2012, 2012 IEEE 12th International Conference on Data Mining.

[20]  Kimiaki Shirahama,et al.  Kobe University and Muroran Institute of Technology at TRECVID 20112 Semantic Indexing Task , 2012, TRECVID.

[21]  Rosalind W. Picard,et al.  Empatica E3 — A wearable wireless multi-sensor device for real-time computerized biofeedback and data acquisition , 2014, 2014 4th International Conference on Wireless Mobile Communication and Healthcare - Transforming Healthcare Through Innovations in Mobile and Wireless Technologies (MOBIHEALTH).

[22]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.

[23]  Yoshua Bengio,et al.  Learning deep physiological models of affect , 2013, IEEE Computational Intelligence Magazine.

[24]  Chong-Wah Ngo,et al.  Representations of Keypoint-Based Semantic Concept Detection: A Comprehensive Study , 2010, IEEE Transactions on Multimedia.

[25]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[26]  Tom Armstrong,et al.  Using Modified Multivariate Bag-of-Words Models to Classify Physiological Data , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[27]  Yuan Li,et al.  Finding Structural Similarity in Time Series Data Using Bag-of-Patterns Representation , 2009, SSDBM.

[28]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[29]  Kimiaki Shirahama,et al.  Hybrid negative example selection using visual and conceptual features , 2011, Multimedia Tools and Applications.

[30]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[31]  Jin-Hyuk Hong,et al.  Understanding physiological responses to stressors during physical activity , 2012, UbiComp.

[32]  George C. Runger,et al.  A Bag-of-Features Framework to Classify Time Series , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Saeid Nahavandi,et al.  Bag-of-words representation for biomedical time series classification , 2012, Biomed. Signal Process. Control..

[34]  Ruzena Bajcsy,et al.  Berkeley MHAD: A comprehensive Multimodal Human Action Database , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[35]  Bernt Schiele,et al.  Scalable Recognition of Daily Activities with Wearable Sensors , 2007, LoCA.

[36]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Cordelia Schmid,et al.  Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[39]  Mi Zhang,et al.  Motion primitive-based human activity recognition using a bag-of-features approach , 2012, IHI '12.

[40]  Kai Kunze,et al.  Smart Eyewear for Interaction and Activity Recognition , 2015, CHI Extended Abstracts.

[41]  Yuan Li,et al.  Rotation-invariant similarity in time series using bag-of-patterns representation , 2012, Journal of Intelligent Information Systems.

[42]  Andrew Zisserman,et al.  The devil is in the details: an evaluation of recent feature encoding methods , 2011, BMVC.

[43]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[44]  Emre Ertin,et al.  Continuous inference of psychological stress from sensory measurements collected in the natural environment , 2011, Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks.

[45]  Yoshua Bengio,et al.  Practical Recommendations for Gradient-Based Training of Deep Architectures , 2012, Neural Networks: Tricks of the Trade.

[46]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[47]  Zhen Wang,et al.  uWave: Accelerometer-based Personalized Gesture Recognition and Its Applications , 2009, PerCom.