Personalizing Activity Recognition Models Through Quantifying Different Types of Uncertainty Using Wearable Sensors

Recognizing activities of daily living (ADL) provides vital contextual information that enhances the effectiveness of various mobile health and wellness applications. Development of wearable motion sensors along with machine learning algorithms offer a great opportunity for ADL recognition. However, the performance of the ADL recognition systems may significantly degrade when they are used by a new user due to inter-subject variability. This issue limits the usability of these systems. In this paper, we propose a deep learning assisted personalization framework for ADL recognition with the aim to maximize the personalization performance while minimizing solicitation of inputs or labels from the user to reduce user's burden. The proposed framework consists of unsupervised retraining of automatic feature extraction layers and supervised fine-tuning of classification layers through a novel active learning model based on a given model's uncertainty. We design a Bayesian deep convolutional neural network with stochastic latent variables that allows us to estimate both aleatoric (data-dependent) and epistemic (model-dependent) uncertainties in recognition task. In this study, for the first time, we show how distinguishing between the two aforementioned sources of uncertainty leads to more effective active learning. The experimental results show that our proposed method improves the accuracy of ADL recognition on a new user by 25% on average compared to the case of using a model for a new user with no personalization with an average final accuracy of 89.2%. Moreover, our method achieves higher personalization accuracy while significantly reducing user's burden in terms of soliciting inputs and labels compared to other methods.

[1]  Roozbeh Jafari,et al.  A human-centered wearable sensing platform with intelligent automated data annotation capabilities , 2019, IoTDI.

[2]  Timo Sztyler,et al.  Unsupervised recognition of interleaved activities of daily living through ontological and probabilistic reasoning , 2016, UbiComp.

[3]  Roozbeh Jafari,et al.  Transferring Activity Recognition Models for New Wearable Sensors with Deep Generative Domain Adaptation , 2019, 2019 18th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[4]  Lei Liu,et al.  Human Daily Activity Recognition for Healthcare Using Wearable and Visual Sensing Data , 2016, 2016 IEEE International Conference on Healthcare Informatics (ICHI).

[5]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[6]  Juha Röning,et al.  Importance of user inputs while using incremental learning to personalize human activity recognition models , 2019, ESANN.

[7]  Andrea Mannini,et al.  Classifier Personalization for Activity Recognition Using Wrist Accelerometers , 2019, IEEE Journal of Biomedical and Health Informatics.

[8]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[9]  Hwee Pink Tan,et al.  Mobile big data analytics using deep learning and apache spark , 2016, IEEE Network.

[10]  Diane J Cook,et al.  Assessing the Quality of Activities in a Smart Environment , 2009, Methods of Information in Medicine.

[11]  Juha Röning,et al.  Personalizing human activity recognition models using incremental learning , 2019, ESANN.

[12]  Timo Sztyler,et al.  Online personalization of cross-subjects based activity recognition models on wearable devices , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[13]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[14]  Gary M. Weiss,et al.  The Impact of Personalization on Smartphone-Based Activity Recognition , 2012, AAAI 2012.

[15]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[16]  Hassan Ghasemzadeh,et al.  Distributed Continuous Action Recognition Using a Hidden Markov Model in Body Sensor Networks , 2009, DCOSS.

[17]  Hassan Ghasemzadeh,et al.  Personalization without User Interruption: Boosting Activity Recognition in New Subjects Using Unlabeled Data , 2017, 2017 ACM/IEEE 8th International Conference on Cyber-Physical Systems (ICCPS).

[18]  Paul J. M. Havinga,et al.  Activity Recognition Using Inertial Sensing for Healthcare, Wellbeing and Sports Applications: A Survey , 2010, ARCS Workshops.

[19]  Sian Lun Lau,et al.  Supporting patient monitoring using activity recognition with a smartphone , 2010, 2010 7th International Symposium on Wireless Communication Systems.

[20]  Mengjie Zhang,et al.  Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptation , 2016, ECCV.

[21]  Roozbeh Jafari,et al.  Hierarchical Signal Segmentation and Classification for Accurate Activity Recognition , 2018, UbiComp/ISWC Adjunct.

[22]  Didier Stricker,et al.  Introducing a New Benchmarked Dataset for Activity Monitoring , 2012, 2012 16th International Symposium on Wearable Computers.

[23]  Nikolaos G. Bourbakis,et al.  A Survey on Wearable Sensor-Based Systems for Health Monitoring and Prognosis , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[24]  Roozbeh Jafari,et al.  A Decision Level Fusion and Signal Analysis Technique for Activity Segmentation and Recognition on Smart Phones , 2018, UbiComp/ISWC Adjunct.

[25]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[26]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[27]  Daniel Roggen,et al.  Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations , 2016, SEMWEB.

[28]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[29]  A. Flammini,et al.  IMU-based solution for automatic detection and classification of exercises in the fitness scenario , 2017, 2017 IEEE Sensors Applications Symposium (SAS).

[30]  Sajal K. Das,et al.  Multimodal Wearable Sensing for Fine-Grained Activity Recognition in Healthcare , 2015, IEEE Internet Computing.

[31]  Fanglin Chen,et al.  StudentLife: assessing mental health, academic performance and behavioral trends of college students using smartphones , 2014, UbiComp.

[32]  Niall Twomey,et al.  Active transfer learning for activity recognition , 2016, ESANN.

[33]  Roozbeh Jafari,et al.  A Deep Learning Assisted Method for Measuring Uncertainty in Activity Recognition with Wearable Sensors , 2019, 2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[34]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[35]  Enamul Hoque,et al.  AALO: Activity recognition in smart homes using Active Learning in the presence of Overlapped activities , 2012, 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops.

[36]  Tahmina Zebin,et al.  Human activity recognition with inertial sensors using a deep learning approach , 2016, 2016 IEEE SENSORS.

[37]  Ling Chen,et al.  Wearable sensor based multimodal human activity recognition exploiting the diversity of classifier ensemble , 2016, UbiComp.

[38]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[39]  Zhijing Liu,et al.  Human Action Recognition Based on Non-linear SVM Decision Tree , 2011 .

[40]  Lina Yao,et al.  Learning from less for better: semi-supervised activity recognition via shared structure discovery , 2016, UbiComp.

[41]  Hassan Ghasemzadeh,et al.  Action coverage formulation for power optimization in body sensor networks , 2008, 2008 Asia and South Pacific Design Automation Conference.

[42]  Yiqiang Chen,et al.  Cross-People Mobile-Phone Based Activity Recognition , 2011, IJCAI.

[43]  R. Venkatesh Babu,et al.  Confidence estimation in Deep Neural networks via density modelling , 2017, ArXiv.

[44]  Danail Stoyanov,et al.  Ambient and Wearable Sensor Fusion for Activity Recognition in Healthcare Monitoring Systems , 2007, BSN.

[45]  Nirmalya Roy,et al.  DeActive , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[46]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[47]  Sivan Sabato,et al.  Interactive Algorithms: from Pool to Stream , 2016, COLT.

[48]  Diane J. Cook,et al.  Designing and evaluating active learning methods for activity recognition , 2014, UbiComp Adjunct.

[49]  Hongnian Yu,et al.  Elderly activities recognition and classification for applications in assisted living , 2013, Expert Syst. Appl..

[50]  Roozbeh Jafari,et al.  Orientation Independent Activity/Gesture Recognition Using Wearable Motion Sensors , 2019, IEEE Internet of Things Journal.

[51]  Tae-Seong Kim,et al.  Smoking Activity Recognition Using a Single Wrist IMU and Deep Learning Light , 2018, ICDSP.

[52]  Ivan Marsic,et al.  Concurrent Activity Recognition with Multimodal CNN-LSTM Structure , 2017, ArXiv.

[53]  Roozbeh Jafari,et al.  MotionSynthesis Toolset (MoST): An Open Source Tool and Data Set for Human Motion Data Synthesis and Validation , 2016, IEEE Sensors Journal.