Facilitating Human Activity Data Annotation via Context-Aware Change Detection on Smartwatches

Annotating activities of daily living (ADL) is vital for developing machine learning models for activity recognition. In addition, it is critical for self-reporting purposes such as in assisted living where the users are asked to log their ADLs. However, data annotation becomes extremely challenging in real-world data collection scenarios, where the users have to provide annotations and labels on their own. Methods such as self-reports that rely on users’ memory and compliance are prone to human errors and become burdensome since they increase users’ cognitive load. In this article, we propose a light yet effective context-aware change point detection algorithm that is implemented and run on a smartwatch for facilitating data annotation for high-level ADLs. The proposed system detects the moments of transition from one to another activity and prompts the users to annotate their data. We leverage freely available Bluetooth low energy (BLE) information broadcasted by various devices to detect changes in environmental context. This contextual information is combined with a motion-based change point detection algorithm, which utilizes data from wearable motion sensors, to reduce the false positives and enhance the system's accuracy. Through real-world experiments, we show that the proposed system improves the quality and quantity of labels collected from users by reducing human errors while eliminating users’ cognitive load and facilitating the data annotation process.

[1]  Sajal K. Das,et al.  HuMAn: Complex Activity Recognition with Multi-Modal Multi-Positional Body Sensing , 2019, IEEE Transactions on Mobile Computing.

[2]  Chris D. Nugent,et al.  Sensor-Based Change Detection for Timely Solicitation of User Engagement , 2017, IEEE Transactions on Mobile Computing.

[3]  Sivan Sabato,et al.  Interactive Algorithms: from Pool to Stream , 2016, COLT.

[4]  Timo Sztyler,et al.  Online personalization of cross-subjects based activity recognition models on wearable devices , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[5]  O. Svensson,et al.  Health-related quality of life and self-reported ability concerning ADL and IADL after hip fracture: A randomized trial , 2006, Acta orthopaedica.

[6]  Chris D. Nugent,et al.  Evaluation of Prompted Annotation of Activity Data Recorded from a Smart Phone , 2014, Sensors.

[7]  Roozbeh Jafari,et al.  Hierarchical Signal Segmentation and Classification for Accurate Activity Recognition , 2018, UbiComp/ISWC Adjunct.

[8]  Niall Twomey,et al.  Active transfer learning for activity recognition , 2016, ESANN.

[9]  Xiangliang Zhang,et al.  A PCA-Based Change Detection Framework for Multidimensional Data Streams: Change Detection in Multidimensional Data Streams , 2015, KDD.

[10]  KawaharaYoshinobu,et al.  Sequential change-point detection based on direct density-ratio estimation , 2012 .

[11]  Roozbeh Jafari,et al.  MotionSynthesis Toolset (MoST): An Open Source Tool and Data Set for Human Motion Data Synthesis and Validation , 2016, IEEE Sensors Journal.

[12]  Paul Lukowicz,et al.  Collecting complex activity datasets in highly rich networked sensor environments , 2010, 2010 Seventh International Conference on Networked Sensing Systems (INSS).

[13]  Samaneh Aminikhanghahi,et al.  Real-Time Change Point Detection with Application to Smart Home Time Series Data , 2019, IEEE Transactions on Knowledge and Data Engineering.

[14]  Hani Hagras,et al.  Autonomous computational intelligence-based behaviour recognition in security and surveillance , 2018, Security + Defence.

[15]  Roozbeh Jafari,et al.  Transferring Activity Recognition Models for New Wearable Sensors with Deep Generative Domain Adaptation , 2019, 2019 18th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[16]  Maureen Schmitter-Edgecombe,et al.  Context-Aware Delivery of Ecological Momentary Assessment , 2020, IEEE Journal of Biomedical and Health Informatics.

[17]  Jun Hu,et al.  Activity recognition based on inertial sensors for Ambient Assisted Living , 2016, 2016 19th International Conference on Information Fusion (FUSION).

[18]  M. Levandowsky,et al.  Distance between Sets , 1971, Nature.

[19]  Masashi Sugiyama,et al.  Change-Point Detection in Time-Series Data by Direct Density-Ratio Estimation , 2009, SDM.

[20]  J. Kristensson,et al.  Prevalence and predictors of healthcare utilization among older people (60+): focusing on ADL dependency and risk of depression. , 2012, Archives of gerontology and geriatrics.

[21]  Mazda A. Marvasti,et al.  Cusum techniques for timeslot sequences with applications to network surveillance , 2009, Comput. Stat. Data Anal..

[22]  Roozbeh Jafari,et al.  A human-centered wearable sensing platform with intelligent automated data annotation capabilities , 2019, IoTDI.

[23]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[24]  Dewen Wang,et al.  Changes in activities of daily living (ADL) among elderly Chinese by marital status, living arrangement, and availability of healthcare over a 3-year period , 2009, Environmental health and preventive medicine.

[25]  Chris D. Nugent,et al.  Online Change Detection for Timely Solicitation of User Interaction , 2014, UCAmI.

[26]  Masashi Sugiyama,et al.  Sequential change‐point detection based on direct density‐ratio estimation , 2012, Stat. Anal. Data Min..

[27]  Ludmila I. Kuncheva,et al.  PCA Feature Extraction for Change Detection in Multidimensional Unlabeled Data , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Nigel Collier,et al.  Change-Point Detection in Time-Series Data by Relative Density-Ratio Estimation , 2012, Neural Networks.

[29]  Roozbeh Jafari,et al.  Orientation Independent Activity/Gesture Recognition Using Wearable Motion Sensors , 2019, IEEE Internet of Things Journal.

[30]  Lin Wang,et al.  The University of Sussex-Huawei Locomotion and Transportation Dataset for Multimodal Analytics With Mobile Devices , 2018, IEEE Access.

[31]  Francesca Bovolo,et al.  Semisupervised One-Class Support Vector Machines for Classification of Remote Sensing Data , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[32]  Sanjay Ranka,et al.  Statistical change detection for multi-dimensional data , 2007, KDD '07.

[33]  Edward R. Sykes,et al.  Context-aware mobile apps using iBeacons: towards smarter interactions , 2015, CASCON.

[34]  Diane J. Cook,et al.  A survey of methods for time series change point detection , 2017, Knowledge and Information Systems.

[35]  Roozbeh Jafari,et al.  Personalizing Activity Recognition Models Through Quantifying Different Types of Uncertainty Using Wearable Sensors , 2020, IEEE Transactions on Biomedical Engineering.

[36]  Roozbeh Jafari,et al.  Using Intelligent Personal Annotations to Improve Human Activity Recognition for Movements in Natural Environments , 2020, IEEE Journal of Biomedical and Health Informatics.

[37]  Deborah Estrin,et al.  Using mobile phones to determine transportation modes , 2010, TOSN.