LabelSens: enabling real-time sensor data labelling at the point of collection using an artificial intelligence-based approach

In recent years, machine learning has developed rapidly, enabling the development of applications with high levels of recognition accuracy relating to the use of speech and images. However, other types of data to which these models can be applied have not yet been explored as thoroughly. Labelling is an indispensable stage of data pre-processing that can be particularly challenging, especially when applied to single or multi-model real-time sensor data collection approaches. Currently, real-time sensor data labelling is an unwieldy process, with a limited range of tools available and poor performance characteristics, which can lead to the performance of the machine learning models being compromised. In this paper, we introduce new techniques for labelling at the point of collection coupled with a pilot study and a systematic performance comparison of two popular types of deep neural networks running on five custom built devices and a comparative mobile app (68.5–89% accuracy within-device GRU model, 92.8% highest LSTM model accuracy). These devices are designed to enable real-time labelling with various buttons, slide potentiometer and force sensors. This exploratory work illustrates several key features that inform the design of data collection tools that can help researchers select and apply appropriate labelling techniques to their work. We also identify common bottlenecks in each architecture and provide field tested guidelines to assist in building adaptive, high-performance edge solutions.

[1]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[2]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[3]  Xavier Serra,et al.  Multi-Label Music Genre Classification from Audio, Text and Images Using Deep Features , 2017, ISMIR.

[4]  Ari Rappoport,et al.  Enhanced Sentiment Learning Using Twitter Hashtags and Smileys , 2010, COLING.

[5]  Wenpeng Yin,et al.  Comparative Study of CNN and RNN for Natural Language Processing , 2017, ArXiv.

[6]  Keith Owen Yeates,et al.  Healthcare at Your Fingertips: The Professional Ethics of Smartphone Health-Monitoring Applications , 2017 .

[7]  Eman M. G. Younis,et al.  Designing and evaluating mobile self-reporting techniques: crowdsourcing for citizen science , 2019, Personal and Ubiquitous Computing.

[8]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[9]  Chee Siang Ang,et al.  NotiMind: Utilizing Responses to Smart Phone Notifications as Affective Sensors , 2017, IEEE Access.

[10]  Bernt Schiele,et al.  Towards Less Supervision in Activity Recognition from Wearable Sensors , 2006, 2006 10th IEEE International Symposium on Wearable Computers.

[11]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[12]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[13]  Youngmoo E. Kim,et al.  MoodSwings: A Collaborative Game for Music Mood Label Collection , 2008, ISMIR.

[14]  Eman M. G. Younis,et al.  NeuroPlace: Categorizing urban places according to mental states , 2017, PloS one.

[15]  Henry A. Kautz,et al.  Real-time crowd labeling for deployable activity recognition , 2013, CSCW.

[16]  Christos Diou,et al.  Image annotation using clickthrough data , 2009, CIVR '09.

[17]  Roger B. Dannenberg,et al.  TagATune: A Game for Music and Sound Annotation , 2007, ISMIR.

[18]  José Bravo,et al.  Ambient Intelligence for Health: First International Conference, AmIHEALTH 2015, Puerto Varas, Chile, December 1-4, 2015, Proceedings , 2015, AmIHEALTH.

[19]  Wojciech Zaremba,et al.  An Empirical Exploration of Recurrent Network Architectures , 2015, ICML.

[20]  Minyi Guo,et al.  Emoticon Smoothed Language Models for Twitter Sentiment Analysis , 2012, AAAI.

[21]  Ali Farhadi,et al.  Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding , 2016, ECCV.

[22]  Yu-chin Tai,et al.  Automatic Road Anomaly Detection Using Smart Mobile Device , 2010 .

[23]  Hiroshi Ishii,et al.  Emerging frameworks for tangible user interfaces , 2000, IBM Syst. J..

[24]  Federica Cena,et al.  Engaging Users in Self-Reporting Their Data: A Tangible Interface for Quantified Self , 2015, HCI.

[25]  Federico Sarzotti,et al.  Self-Monitoring of Emotions and Mood Using a Tangible Approach , 2018, Comput..

[26]  Eman M. G. Younis,et al.  Towards unravelling the relationship between on-body, environmental and emotion data using sensor information fusion approach , 2018, Inf. Fusion.

[27]  Eman M. G. Younis,et al.  Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection , 2019, Inf. Fusion.

[28]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[29]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[30]  Jennifer Wortman Vaughan.  Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research , 2017, J. Mach. Learn. Res..

[31]  Lukasz Kaiser,et al.  Neural GPUs Learn Algorithms , 2015, ICLR.

[32]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[33]  Yoshua Bengio,et al.  Generative Adversarial Networks , 2014, ArXiv.

[34]  Eiman Kanjo,et al.  NoiseSPY: A Real-Time Mobile Phone Platform for Urban Noise Monitoring and Mapping , 2010, Mob. Networks Appl..

[35]  Kent Larson,et al.  Real-Time Recognition of Physical Activities and Their Intensities Using Wireless Accelerometers and a Heart Rate Monitor , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[36]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.