Learning multi-level features for sensor-based human action recognition

Abstract This paper proposes a multi-level feature learning framework for human action recognition using a single body-worn inertial sensor. The framework consists of three phases, respectively designed to analyze signal-based (low-level), components (mid-level) and semantic (high-level) information. Low-level features capture the time and frequency domain property while mid-level representations learn the composition of the action. The Max-margin Latent Pattern Learning (MLPL) method is proposed to learn high-level semantic descriptions of latent action patterns as the output of our framework. The proposed method achieves the state-of-the-art performances, 88.7%, 98.8% and 72.6% (weighted F 1 score) respectively, on Skoda, WISDM and OPP datasets.

[1]  Jiang Zhu,et al.  Mobile Lifelogger - Recording, Indexing, and Understanding a Mobile User's Life , 2010, MobiCASE.

[2]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[3]  Alberto Calatroni,et al.  Improving online gesture recognition with template matching methods in accelerometer data , 2012, 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA).

[4]  Lian-Wen Jin,et al.  Activity recognition from acceleration data using AR model representation and SVM , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[5]  Michael L. Littman,et al.  Activity Recognition from Accelerometer Data , 2005, AAAI.

[6]  Bernt Schiele,et al.  Multi Activity Recognition Based on Bodymodel-Derived Primitives , 2009, LoCA.

[7]  Shengrui Wang,et al.  A Frequent Pattern Mining Approach for ADLs Recognition in Smart Environments , 2011, 2011 IEEE International Conference on Advanced Information Networking and Applications.

[8]  Jian Lu,et al.  An unsupervised approach to activity recognition and segmentation based on object-use fingerprints , 2010, Data Knowl. Eng..

[9]  Bernt Schiele,et al.  Analyzing features for activity recognition , 2005, sOc-EUSAI '05.

[10]  Zhenyu He,et al.  Activity recognition from acceleration data based on discrete consine transform and SVM , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[11]  Montserrat Ros,et al.  Recognizing human motions through mixture modeling of inertial data , 2015, Pattern Recognit..

[12]  Diane J. Cook,et al.  Human Activity Recognition and Pattern Discovery , 2010, IEEE Pervasive Computing.

[13]  A. Belousov,et al.  A flexible classification approach with optimal generalisation performance: support vector machines , 2002 .

[14]  Mi Zhang,et al.  Motion primitive-based human activity recognition using a bag-of-features approach , 2012, IHI '12.

[15]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[16]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[18]  Seungjin Choi,et al.  Multi-modal Convolutional Neural Networks for Activity Recognition , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[19]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[20]  Alberto Calatroni,et al.  Robust online gesture recognition with crowdsourced annotations , 2014, J. Mach. Learn. Res..

[21]  Qiang Ji,et al.  Modeling Temporal Interactions with Interval Temporal Bayesian Networks for Complex Activity Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..

[23]  Yasushi Makihara,et al.  Similar gait action recognition using an inertial sensor , 2015, Pattern Recognit..

[24]  Bernt Schiele,et al.  Discovery of activity patterns using topic models , 2008 .

[25]  Ricardo Chavarriaga,et al.  The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition , 2013, Pattern Recognit. Lett..

[26]  Tao Gu,et al.  Object relevance weight pattern mining for activity recognition and segmentation , 2010, Pervasive Mob. Comput..

[27]  Alexei A. Efros,et al.  Ensemble of exemplar-SVMs for object detection and beyond , 2011, 2011 International Conference on Computer Vision.

[28]  Thomas Plötz,et al.  Using unlabeled data in a sparse-coding framework for human activity recognition , 2014, Pervasive Mob. Comput..

[29]  Bernt Schiele,et al.  ADL recognition based on the combination of RFID and accelerometer sensing , 2008, 2008 Second International Conference on Pervasive Computing Technologies for Healthcare.

[30]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[31]  Tao Xiang,et al.  Identifying Rare and Subtle Behaviors: A Weakly Supervised Joint Topic Model , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Robert Bergevin,et al.  Semantic human activity recognition: A literature review , 2015, Pattern Recognit..

[33]  Christian Vollmer,et al.  Learning Features for Activity Recognition with Shift-Invariant Sparse Coding , 2013, ICANN.

[34]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.

[35]  Yang Yi,et al.  Human action recognition with graph-based multiple-instance learning , 2016, Pattern Recognit..

[36]  Ying Zhang,et al.  SensCare: Semi-automatic Activity Summarization System for Elderly Care , 2011, MobiCASE.

[37]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[38]  Paul Lukowicz,et al.  Collecting complex activity datasets in highly rich networked sensor environments , 2010, 2010 Seventh International Conference on Networked Sensing Systems (INSS).

[39]  T. Kuo,et al.  The application of cepstral coefficients and maximum likelihood method in EMG pattern recognition. , 1995, IEEE transactions on bio-medical engineering.

[40]  Bernt Schiele,et al.  Activity Recognition from Sparsely Labeled Data Using Multi-Instance Learning , 2009, LoCA.

[41]  David S. Rosenblum,et al.  From action to activity: Sensor-based activity recognition , 2016, Neurocomputing.

[42]  Peter Andras,et al.  On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution , 2013, ISWC '13.

[43]  Davide Anguita,et al.  Human Activity Recognition on Smartphones with Awareness of Basic Activities and Postural Transitions , 2014, ICANN.

[44]  Claudio Gallicchio,et al.  Human activity recognition using multisensor data fusion based on Reservoir Computing , 2016, J. Ambient Intell. Smart Environ..

[45]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[46]  Bernt Schiele,et al.  Daily Routine Recognition through Activity Spotting , 2009, LoCA.

[47]  Henry Kautz,et al.  Hierarchical recognition of activities of daily living using multi-scale, multi-perspective vision and RFID , 2008 .

[48]  Zhuowen Tu,et al.  Max-Margin Multiple-Instance Dictionary Learning , 2013, ICML.

[49]  Bernt Schiele,et al.  Weakly Supervised Recognition of Daily Life Activities with Wearable Sensors , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  T Togawa,et al.  Classification of acceleration waveforms during walking by wavelet transform. , 1997, Methods of information in medicine.

[51]  Jiang Zhu,et al.  MobiSens: A Versatile Mobile Sensing Platform for Real-World Applications , 2013, Mob. Networks Appl..

[52]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[53]  Norbert Link,et al.  Gesture recognition with inertial sensors and optimized DTW prototypes , 2010, 2010 IEEE International Conference on Systems, Man and Cybernetics.

[54]  Patrick Olivier,et al.  Feature Learning for Activity Recognition in Ubiquitous Computing , 2011, IJCAI.

[55]  Paul Lukowicz,et al.  Wearable Activity Tracking in Car Manufacturing , 2008, IEEE Pervasive Computing.

[56]  Gerhard Tröster,et al.  Probabilistic parsing of dietary activity events , 2007, BSN.

[57]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[58]  Bin Gu,et al.  Data Sparseness in Linear SVM , 2015, IJCAI.

[59]  Svetha Venkatesh,et al.  The Hidden Permutation Model and Location-Based Activity Recognition , 2008, AAAI.

[60]  Jorge Dias,et al.  Trajectory-based human action segmentation , 2015, Pattern Recognit..

[61]  Luca Benini,et al.  Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection , 2008, EWSN.

[62]  Bernt Schiele,et al.  Scalable Recognition of Daily Activities with Wearable Sensors , 2007, LoCA.

[63]  Gerhard Tröster,et al.  Unsupervised Classifier Self-Calibration through Repeated Context Occurences: Is there Robustness against Sensor Displacement to Gain? , 2009, 2009 International Symposium on Wearable Computers.