Cross-Dataset Activity Recognition via Adaptive Spatial-Temporal Transfer Learning

Human activity recognition (HAR) aims at recognizing activities by training models on the large quantity of sensor data. Since it is time-consuming and expensive to acquire abundant labeled data, transfer learning becomes necessary for HAR by transferring knowledge from existing domains. However, there are two challenges existing in cross-dataset activity recognition. The first challenge is source domain selection. Given a target task and several available source domains, it is difficult to determine how to select the most similar source domain to the target domain such that negative transfer can be avoided. The second one is accurately activity transfer. After source domain selection, how to achieve accurate knowledge transfer between the selected source and the target domain remains another challenge. In this paper, we propose an Adaptive Spatial-Temporal Transfer Learning (ASTTL) approach to tackle both of the above two challenges in cross-dataset HAR. ASTTL learns the spatial features in transfer learning by adaptively evaluating the relative importance between the marginal and conditional probability distributions. Besides, it captures the temporal features via incremental manifold learning. Therefore, ASTTL can learn the adaptive spatial-temporal features for cross-dataset HAR and can be used for both source domain selection and accurate activity transfer. We evaluate the performance of ASTTL through extensive experiments on 4 public HAR datasets, which demonstrates its effectiveness. Furthermore, based on ASTTL, we design and implement an adaptive cross-dataset HAR system called Client-Cloud Collaborative Adaptive Activity Recognition System (3C2ARS) to perform HAR in the real environment. By collecting activities in the smartphone and transferring knowledge in the cloud server, ASTTL can significantly improve the performance of source domain selection and accurate activity transfer.

[1]  Yiqiang Chen,et al.  Deep Transfer Learning for Cross-domain Activity Recognition , 2018, ICCSE'18.

[2]  Koby Crammer,et al.  Analysis of Representations for Domain Adaptation , 2006, NIPS.

[3]  Qiang Yang,et al.  Action-model acquisition for planning via transfer learning , 2014, Artif. Intell..

[4]  Nirmalya Roy,et al.  Active learning enabled activity recognition , 2016, 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[5]  Ian J. Wassell,et al.  Re-weighted Adversarial Adaptation Network for Unsupervised Domain Adaptation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Min Chen,et al.  Study on Driving Fatigue Evaluation System Based on Short Time Period ECG Signal , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[7]  Davide Anguita,et al.  Human Activity Recognition on Smartphones Using a Multiclass Hardware-Friendly Support Vector Machine , 2012, IWAAL.

[8]  Philip S. Yu,et al.  Transfer Feature Learning with Joint Distribution Adaptation , 2013, 2013 IEEE International Conference on Computer Vision.

[9]  Qiang Yang,et al.  Distant Domain Transfer Learning , 2017, AAAI.

[10]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[11]  Dianhui Chu,et al.  Empirical Study and Improvement on Deep Transfer Learning for Human Activity Recognition , 2018, Sensors.

[12]  Yang Song,et al.  Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Robert Remus,et al.  Domain Adaptation Using Domain Similarity- and Domain Complexity-Based Instance Selection for Cross-Domain Sentiment Analysis , 2012, 2012 IEEE 12th International Conference on Data Mining Workshops.

[14]  Yuan Shi,et al.  Geodesic flow kernel for unsupervised domain adaptation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Nirmalya Roy,et al.  DeActive , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[16]  Kate Saenko,et al.  Return of Frustratingly Easy Domain Adaptation , 2015, AAAI.

[17]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[18]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[19]  Fuzhen Zhuang,et al.  Supervised Representation Learning: Transfer Learning with Deep Autoencoders , 2015, IJCAI.

[20]  Le Song,et al.  A Hilbert Space Embedding for Distributions , 2007, Discovery Science.

[21]  Mi Zhang,et al.  USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors , 2012, UbiComp.

[22]  Marco Loog,et al.  Distance Based Source Domain Selection for Sentiment Classification , 2018, ArXiv.

[23]  Jianhua Lin,et al.  Divergence measures based on the Shannon entropy , 1991, IEEE Trans. Inf. Theory.

[24]  Don Coppersmith,et al.  Matrix multiplication via arithmetic progressions , 1987, STOC.

[25]  Archan Misra,et al.  Scaling Human Activity Recognition via Deep Learning-based Domain Adaptation , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[26]  Daniel D. Lee,et al.  Grassmann discriminant analysis: a unifying view on subspace-based learning , 2008, ICML '08.

[27]  Paul Lukowicz,et al.  Label Propagation , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[28]  Bernhard Schölkopf,et al.  A Kernel Two-Sample Test , 2012, J. Mach. Learn. Res..

[29]  Qiang Yang,et al.  Cross Validation Framework to Choose amongst Models and Datasets for Transfer Learning , 2010, ECML/PKDD.

[30]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[31]  Roland Kuhn,et al.  Discriminative Instance Weighting for Domain Adaptation in Statistical Machine Translation , 2010, EMNLP.

[32]  Kate Saenko,et al.  Deep CORAL: Correlation Alignment for Deep Domain Adaptation , 2016, ECCV Workshops.

[33]  Mariella Dimiccoli,et al.  Mitigating Bystander Privacy Concerns in Egocentric Activity Recognition with Deep Learning and Intentional Image Degradation , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[34]  Xin Qin,et al.  FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare , 2019, IEEE Intelligent Systems.

[35]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[36]  Philip S. Yu,et al.  Stratified Transfer Learning for Cross-domain Activity Recognition , 2017, 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[37]  Bernhard Schölkopf,et al.  A Kernel Method for the Two-Sample-Problem , 2006, NIPS.

[38]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[39]  Jian Zhang,et al.  Double-bootstrapping source data selection for instance-based transfer learning , 2013, Pattern Recognit. Lett..

[40]  Philip S. Yu,et al.  Domain Invariant Transfer Kernel Learning , 2015, IEEE Transactions on Knowledge and Data Engineering.

[41]  Rama Chellappa,et al.  Domain adaptation for object recognition: An unsupervised approach , 2011, 2011 International Conference on Computer Vision.

[42]  Qiang Yang,et al.  Cross-domain activity recognition via transfer learning , 2011, Pervasive Mob. Comput..

[43]  Sajal K. Das,et al.  A-Wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[44]  Qiang Yang,et al.  Transfer Learning via Dimensionality Reduction , 2008, AAAI.

[45]  Philip S. Yu,et al.  Visual Domain Adaptation with Manifold Embedded Distribution Alignment , 2018, ACM Multimedia.

[46]  Bernt Schiele,et al.  Discovery of activity patterns using topic models , 2008 .

[47]  Didier Stricker,et al.  Introducing a New Benchmarked Dataset for Activity Monitoring , 2012, 2012 16th International Symposium on Wearable Computers.

[48]  Brian C. Lovell,et al.  Domain Adaptation on the Statistical Manifold , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Dawei Liang,et al.  Audio-Based Activities of Daily Living (ADL) Recognition with Large-Scale Acoustic Embeddings from Online Videos , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[50]  Yiqiang Chen,et al.  Balanced Distribution Adaptation for Transfer Learning , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[51]  Sergiu M. Dascalu,et al.  Spatiotemporal recursive hyperspheric classification with an application to dynamic gesture recognition , 2019, Artif. Intell..

[52]  Yiqiang Chen,et al.  Transfer Learning with Dynamic Adversarial Adaptation Network , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[53]  Hassan Ghasemzadeh,et al.  Autonomous Training of Activity Recognition Algorithms in Mobile Sensors: A Transfer Learning Approach in Context-Invariant Views , 2018, IEEE Transactions on Mobile Computing.

[54]  Billur Barshan,et al.  Recognizing Daily and Sports Activities in Two Open Source Machine Learning Environments Using Body-Worn Sensor Units , 2014, Comput. J..

[55]  Shuangquan Wang,et al.  Extreme learning machine-based device displacement free activity recognition model , 2012, Soft Comput..

[56]  Yiqiang Chen,et al.  Cross-position Activity Recognition with Stratified Transfer Learning , 2018, Pervasive Mob. Comput..

[57]  Jianfei Yu,et al.  Instance-based Domain Adaptation via Multiclustering Logistic Approximation , 2018, IEEE Intelligent Systems.

[58]  John Blitzer,et al.  Domain Adaptation with Structural Correspondence Learning , 2006, EMNLP.

[59]  Tao Gu,et al.  Towards a Diffraction-based Sensing Approach on Human Activity Recognition , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[60]  Yiqiang Chen,et al.  Transfer Learning with Dynamic Distribution Adaptation , 2019, ACM Trans. Intell. Syst. Technol..

[61]  David Wagner,et al.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods , 2017, AISec@CCS.

[62]  Arun Ross,et al.  On automated source selection for transfer learning in convolutional neural networks , 2018, Pattern Recognit..

[63]  Jindong Wang,et al.  Easy Transfer Learning By Exploiting Intra-Domain Structures , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[64]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.