Recognition of Activities of Daily Living and Environments Using Acoustic Sensors Embedded on Mobile Devices

The identification of Activities of Daily Living (ADL) is intrinsic with the user’s environment recognition. This detection can be executed through standard sensors present in every-day mobile devices. On the one hand, the main proposal is to recognize users’ environment and standing activities. On the other hand, these features are included in a framework for the ADL and environment identification. Therefore, this paper is divided into two parts—firstly, acoustic sensors are used for the collection of data towards the recognition of the environment and, secondly, the information of the environment recognized is fused with the information gathered by motion and magnetic sensors. The environment and ADL recognition are performed by pattern recognition techniques that aim for the development of a system, including data collection, processing, fusion and classification procedures. These classification techniques include distinctive types of Artificial Neural Networks (ANN), analyzing various implementations of ANN and choosing the most suitable for further inclusion in the following different stages of the developed system. The results present 85.89% accuracy using Deep Neural Networks (DNN) with normalized data for the ADL recognition and 86.50% accuracy using Feedforward Neural Networks (FNN) with non-normalized data for environment recognition. Furthermore, the tests conducted present 100% accuracy for standing activities recognition using DNN with normalized data, which is the most suited for the intended purpose.

[1]  Hirozumi Yamaguchi,et al.  CrowdMeter: Congestion Level Estimation in Railway Stations Using Smartphones , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[2]  Yong Liu,et al.  Environmental sound recognition using time-frequency intersection patterns , 2011, iCAST.

[3]  Pablo Casaseca-de-la-Higuera,et al.  Efficient k-NN Implementation for Real-Time Detection of Cough Events in Smartphones , 2018, IEEE Journal of Biomedical and Health Informatics.

[4]  Michael Cheffena,et al.  Fall Detection Using Smartphone Audio Features , 2016, IEEE Journal of Biomedical and Health Informatics.

[5]  Nuno M. Garcia,et al.  From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices , 2016, Sensors.

[6]  Nasser Kehtarnavaz,et al.  Real-time implementation of voice activity detector on ARM embedded processor of smartphones , 2017, 2017 IEEE 26th International Symposium on Industrial Electronics (ISIE).

[7]  Gerhard Tröster,et al.  AmbientSense: A real-time ambient sound recognition system for smartphones , 2013, 2013 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[8]  Weihua Sheng,et al.  AutoHydrate: A wearable hydration monitoring system , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Christiane Gresse von Wangenheim,et al.  A Systematic Literature Review on Usability Heuristics for Mobile Phones , 2013, Int. J. Mob. Hum. Comput. Interact..

[10]  Rainer Brück,et al.  Design and evaluation of a smartphone application for non-speech sound awareness for people with hearing loss , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[11]  Thomas George,et al.  An effective approach for human activity recognition on smartphone , 2015, 2015 IEEE International Conference on Engineering and Technology (ICETECH).

[12]  Rainer Brück,et al.  Smartphone application for automatic classification of environmental sound , 2013, Proceedings of the 20th International Conference Mixed Design of Integrated Circuits and Systems - MIXDES 2013.

[13]  Vasileios Bountourakis,et al.  Machine Learning Algorithms for Environmental Sound Recognition: Towards Soundscape Semantics , 2015, AM '15.

[14]  Mi Zhang,et al.  BodyBeat: a mobile system for sensing non-speech body sounds , 2014, MobiSys.

[15]  Nasser Kehtarnavaz,et al.  Smartphone-based real-time classification of noise signals using subband features and random forest classifier , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[16]  Brian Caulfield,et al.  Pervasive Sound Sensing: A Weakly Supervised Training Approach , 2016, IEEE Transactions on Cybernetics.

[17]  Shingo Uenohara,et al.  Snore activity detection using smartphone sensors , 2015, 2015 IEEE International Conference on Consumer Electronics - Taiwan.

[18]  Shingo Uenohara,et al.  A Study on the Optimum Number of Training Data in Snore Activity Detection Using SVM , 2016, 2016 10th International Conference on Complex, Intelligent, and Software Intensive Systems (CISIS).

[19]  Xin Jin,et al.  SmartBuddy: An Integrated Mobile Sensing and Detecting System for Family Activities , 2017, GLOBECOM 2017 - 2017 IEEE Global Communications Conference.

[20]  Flora Amato,et al.  Extreme events management using multimedia social networks , 2019, Future Gener. Comput. Syst..

[21]  Gebremedhin Teklemariam Abreha An Environmental Audio{Based Context Recognition System Using Smartphones , 2014 .

[22]  N. Garcia,et al.  Multi-sensor data fusion techniques for the identification of activities of daily living using mobile devices , 2015 .

[23]  Geoff V. Merrett,et al.  Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence A Hidden Markov Model-Based Acoustic Cicada Detector for , 2022 .

[24]  Yu-Chee Tseng,et al.  Inference of Conversation Partners by Cooperative Acoustic Sensing in Smartphone Networks , 2016, IEEE Transactions on Mobile Computing.

[25]  Chris D. Nugent,et al.  Sensor-Based Activity Recognition Review , 2019, Human Activity Recognition and Behaviour Analysis.

[26]  Fu-Shan Jaw,et al.  Smartphone-based fall detection algorithm using feature extraction , 2016, 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[27]  Andrew T. Campbell,et al.  Bewell: A smartphone application to monitor, model and promote wellbeing , 2011, PervasiveHealth 2011.

[28]  Sotiris E. Nikoletseas,et al.  Hierarchical algorithm for daily activity recognition via smartphone sensors , 2015, 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT).

[29]  Diane J. Cook,et al.  Simple and Complex Activity Recognition through Smart Phones , 2012, 2012 Eighth International Conference on Intelligent Environments.

[30]  Swetha Machanavajhala,et al.  A REAL-TIME ENVIRONMENTAL SOUND RECOGNITION SYSTEM FOR THE ANDROID OS , 2016 .

[31]  Nicholas D. Lane,et al.  DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning , 2015, UbiComp.

[32]  Gianluca Vinti,et al.  Pointwise and uniform approximation by multivariate neural network operators of the max-product type , 2016, Neural Networks.

[33]  Ehsan Valavi,et al.  Microsoft Word-Final.docx , 2010 .

[34]  Yutaka Arakawa,et al.  Approaching vehicle detection method with acoustic analysis using smartphone for elderly bicycle driver , 2017, 2017 Tenth International Conference on Mobile Computing and Ubiquitous Network (ICMU).

[35]  Vincent Fontaine,et al.  Automatic classification of environmental noise events by hidden Markov models , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[36]  Vincenzo Moscato,et al.  An Edge Intelligence Empowered Recommender System Enabling Cultural Heritage Applications , 2019, IEEE Transactions on Industrial Informatics.

[37]  Sacha Krstulovic,et al.  Automatic Environmental Sound Recognition: Performance Versus Computational Cost , 2016, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[38]  Harishchandra Dubey,et al.  BigEAR: Inferring the Ambient and Emotional Correlates from Smartphone-Based Acoustic Big Data , 2016, 2016 IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE).

[39]  Jiguo Yu,et al.  Detecting driver phone calls in a moving vehicle based on voice features , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[40]  Pablo Casaseca-de-la-Higuera,et al.  Robust Detection of Audio-Cough Events Using Local Hu Moments , 2019, IEEE Journal of Biomedical and Health Informatics.

[41]  Yufei Chen,et al.  On motion-sensor behavior analysis for human-activity recognition via smartphones , 2016, 2016 IEEE International Conference on Identity, Security and Behavior Analysis (ISBA).

[42]  Zhu Wang,et al.  Recognition of Human Computer Operations Based on Keystroke Sensing by Smartphone Microphone , 2018, IEEE Internet of Things Journal.

[43]  Jian Wang,et al.  Accurate Combined Keystrokes Detection Using Acoustic Signals , 2016, 2016 12th International Conference on Mobile Ad-Hoc and Sensor Networks (MSN).

[44]  Masafumi Nishida,et al.  Daily activity recognition based on acoustic signals and acceleration signals estimated with Gaussian process , 2015, 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[45]  Nuno M. Garcia,et al.  Identification of Activities of Daily Living Using Sensors Available in off-the-shelf Mobile Devices: Research and Hypothesis , 2016, ISAmI.

[46]  Guoliang Xing,et al.  FamilyLog: A mobile system for monitoring family mealtime activities , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[47]  Naonori Ueda,et al.  Mobile activity recognition for a whole day: recognizing real nursing activities with big dataset , 2015, UbiComp.

[48]  G. Gripenberg,et al.  Approximation by neural networks with a bounded number of nodes at each level , 2003, J. Approx. Theory.

[49]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[50]  Naveen Aggarwal,et al.  Acoustic Scene Classification for Personal Commuting Mode: Detecting Polluting vs. Non Polluting Vehicles , 2018, 2018 8th International Conference on Cloud Computing, Data Science & Engineering (Confluence).

[51]  Ramón F. Brena,et al.  Feature Selection for Place Classification through Environmental Sounds , 2014, EUSPN/ICTH.

[52]  Nuno M. Garcia,et al.  Identification of activities of daily living through data fusion on motion and magnetic sensors embedded on mobile devices , 2018, Pervasive Mob. Comput..

[53]  C. Rader,et al.  A new principle for fast Fourier transformation , 1976 .

[54]  Nuno M. Garcia,et al.  Pattern recognition techniques for the identification of Activities of Daily Living using mobile device accelerometer , 2019, PeerJ Prepr..

[55]  Alípio Mário Jorge,et al.  Using Smartphones to Classify Urban Sounds , 2016, C3S2E.

[56]  DeLiang Wang,et al.  Pattern recognition: neural networks in perspective , 1993, IEEE Expert.

[57]  Nuno M. Garcia A Roadmap to the Design of a Personal Digital Life Coach , 2015, ICT Innovations.

[58]  Héctor Pomares,et al.  On the Use of Sensor Fusion to Reduce the Impact of Rotational and Additive Noise in Human Activity Recognition , 2012, Sensors.

[59]  K. Doya,et al.  Exciting Time for Neural Networks , 2015 .

[60]  Wei Pan,et al.  SoundSense: scalable sound sensing for people-centric applications on mobile phones , 2009, MobiSys '09.

[61]  Kit Simpson,et al.  Activities of Daily Living Item Bank , 2018 .