Continual Activity Recognition with Generative Adversarial Networks

Continual learning is an emerging research challenge in human activity recognition (HAR). As an increasing number of HAR applications are deployed in real-world environments, it is important and essential to extend the activity model to adapt to the change in people’s activity routine. Otherwise, HAR applications can become obsolete and fail to deliver activity-aware services. The existing research in HAR has focused on detecting abnormal sensor events or new activities, however, extending the activity model is currently under-explored. To directly tackle this challenge, we build on the recent advance in the area of lifelong machine learning and design a continual activity recognition system, called HAR-GAN, to grow the activity model over time. HAR-GAN does not require a prior knowledge on what new activity classes might be and it does not require to store historical data by leveraging the use of Generative Adversarial Networks (GAN) to generate sensor data on the previously learned activities. We have evaluated HAR-GAN on four third-party, public datasets collected on binary sensors and accelerometers. Our extensive empirical results demonstrate the effectiveness of HAR-GAN in continual activity recognition and shed insight on the future challenges.

[1]  Lei Fang,et al.  Discovery and Recognition of Emerging Human Activities Using a Hierarchical Mixture of Directional Statistical Models , 2020, IEEE Transactions on Knowledge and Data Engineering.

[2]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[3]  Gabriela Marcu,et al.  Challenges Providing Ground Truth for Pervasive Healthcare Systems , 2019, IEEE Pervasive Computing.

[4]  Ronald Kemker,et al.  Measuring Catastrophic Forgetting in Neural Networks , 2017, AAAI.

[5]  Razvan Pascanu,et al.  Progressive Neural Networks , 2016, ArXiv.

[6]  P. Barry.  Deep , 2018, 2018 11th International Conference on Human System Interaction (HSI).

[7]  Gwenn Englebienne,et al.  Human activity recognition from wireless sensor network data: benchmark and software , 2011 .

[8]  Nitesh V. Chawla,et al.  SMOTE: Synthetic Minority Over-sampling Technique , 2002, J. Artif. Intell. Res..

[9]  Dahua Lin,et al.  Learning a Unified Classifier Incrementally via Rebalancing , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jiwon Kim,et al.  Continual Learning with Deep Generative Replay , 2017, NIPS.

[11]  Franco Zambonelli,et al.  Lifelong Learning in Sensor-Based Human Activity Recognition , 2019, IEEE Pervasive Computing.

[12]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[13]  Nirmalya Roy,et al.  Active learning enabled activity recognition , 2016, 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[14]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[15]  Didier Stricker,et al.  Introducing a New Benchmarked Dataset for Activity Monitoring , 2012, 2012 16th International Symposium on Wearable Computers.

[16]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Surya Ganguli,et al.  Continual Learning Through Synaptic Intelligence , 2017, ICML.

[18]  Kilian Q. Weinberger,et al.  An empirical study on evaluation metrics of generative adversarial networks , 2018, ArXiv.

[19]  Philip S. Yu,et al.  Stratified Transfer Learning for Cross-domain Activity Recognition , 2017, 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[20]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Christoph H. Lampert,et al.  iCaRL: Incremental Classifier and Representation Learning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Stefan Wermter,et al.  Continual Lifelong Learning with Neural Networks: A Review , 2019, Neural Networks.

[23]  Franco Zambonelli,et al.  XLearn: Learning Activity Labels across Heterogeneous Datasets , 2020, ACM Trans. Intell. Syst. Technol..

[24]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[27]  Andreas S. Tolias,et al.  Generative replay with feedback connections as a general strategy for continual learning , 2018, ArXiv.

[28]  Hristijan Gjoreski,et al.  Unsupervised online activity discovery using temporal behaviour assumption , 2017, SEMWEB.

[29]  Praminda Caleb-Solly,et al.  Unsupervised Machine Learning for Developing Personalised Behaviour Models Using Activity Data , 2017, Sensors.

[30]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[31]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[32]  Martin L. Griss,et al.  NuActiv: recognizing unseen new activities using semantic attribute-based learning , 2013, MobiSys '13.

[33]  R. French.  Catastrophic forgetting in connectionist networks , 1999, Trends in Cognitive Sciences.

[34]  Andreas S. Tolias,et al.  Three scenarios for continual learning , 2019, ArXiv.

[35]  Nicolas Y. Masse,et al.  Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization , 2018, Proceedings of the National Academy of Sciences.

[36]  Derek Hoiem,et al.  Learning without Forgetting , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Lina Yao,et al.  Converting Your Thoughts to Texts: Enabling Brain Typing via Deep Feature Learning of EEG Signals , 2017, 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[38]  Sebastian Nowozin,et al.  Which Training Methods for GANs do actually Converge? , 2018, ICML.

[39]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[40]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[41]  Jonathon Shlens,et al.  Conditional Image Synthesis with Auxiliary Classifier GANs , 2016, ICML.

[42]  Bala Srinivasan,et al.  AnyNovel: detection of novel concepts in evolving data streams , 2016, Evolving Systems.

[43]  Billur Barshan,et al.  Recognizing Daily and Sports Activities in Two Open Source Machine Learning Environments Using Body-Worn Sensor Units , 2014, Comput. J..

[44]  L. Carin,et al.  Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation , 2019, ACCV.

[45]  VALENTIN RADU,et al.  Multimodal Deep Learning for Activity and Context Recognition , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[46]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[47]  Sozo Inoue,et al.  Position independent activity recognition using shallow neural architecture and empirical modeling , 2019, UbiComp/ISWC Adjunct.

[48]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[49]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[50]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Bing Liu,et al.  Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation , 2018, ICLR.

[52]  W. Hager,et al.  and s , 2019, Shallow Water Hydraulics.

[53]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[54]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[55]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).