Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations

Kernels in the convolutional layers of deep convolutional networks are believed to act as feature extractors, progressively highlighting more domain-specific features in the upper network layers. Thus lower-level features might be suitable for transfer. We analyse this in wearable activity recognition by reusing kernels learned on a source domain on another target domain. We consider transfer between users, application domains, sensor modalities and sensor locations. We characterize the trade-offs of transferring various convolutional layers along model size, learning speed, recognition performance and training data. Through novel kernel visualisations and comparative evaluations we identify what kernels are predominantly sensitive to, amongst sensor characteristics, motion dynamics and on-body placement. Kernel transfer reduces training time by ~17% without additional complexity. We derive recommendations on when transfer is most suitable.

[1]  Qiang Yang,et al.  Cross-domain activity recognition , 2009, UbiComp.

[2]  Jürgen Schmidhuber,et al.  Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction , 2011, ICANN.

[3]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[4]  Paul Lukowicz,et al.  Collecting complex activity datasets in highly rich networked sensor environments , 2010, 2010 Seventh International Conference on Networked Sensing Systems (INSS).

[5]  Luca Benini,et al.  Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection , 2008, EWSN.

[6]  Gwenn Englebienne,et al.  In-Home Activity Recognition: Bayesian Inference for Hidden Markov Models , 2014, IEEE Pervasive Computing.

[7]  Benjamin Schrauwen,et al.  End-to-end learning for music audio , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Thomas Plötz,et al.  Using unlabeled data in a sparse-coding framework for human activity recognition , 2014, Pervasive Mob. Comput..

[9]  Kwang-Ting Cheng,et al.  Using mobile GPU for general-purpose computing – a case study of face recognition on smartphones , 2011, Proceedings of 2011 International Symposium on VLSI Design, Automation and Test.

[10]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[11]  Stefan Winkler,et al.  Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning , 2015, ICMI.

[12]  Ricardo Chavarriaga,et al.  Dynamic Quantification of Activity Recognition Capabilities in Opportunistic Systems , 2011, 2011 IEEE 73rd Vehicular Technology Conference (VTC Spring).

[13]  Daniel Roggen,et al.  Automatic Transfer of Activity Recognition Capabilities between Body-Worn Motion Sensors: Training Newcomers to Recognize Locomotion , 2011 .

[14]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Diane J. Cook,et al.  Transfer learning for activity recognition: a survey , 2013, Knowledge and Information Systems.

[17]  Alois Ferscha,et al.  Real-Time Transfer and Evaluation of Activity Recognition Capabilities in an Opportunistic System , 2011 .

[18]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[19]  Michael I. Jordan,et al.  Learning Transferable Features with Deep Adaptation Networks , 2015, ICML.

[20]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[21]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[22]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.