Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion

Abstract More and more common activities are leading to a sedentary lifestyle forcing us to sit several hours every day. In-seat actions contain significant hidden information, which not only reflects the current physical health status but also can report mental states. Considering this, we design a system, based on body-worn inertial sensors (attached to user’s wrists) combined with a pressure detection module (deployed on the seat), to recognise and monitor in-seat activities through sensor- and feature-level fusion techniques. Specifically, we focus on four common basic emotion-relevant activities (i.e. interest-, frustration-, sadness- and happiness-related). Our results show that the proposed method, by fusion of time- and frequency-domain feature sets from all the different deployed sensors, can achieve high accuracy in recognising the considered activities.

[1]  Hassan Ghasemzadeh,et al.  Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges , 2017, Inf. Fusion.

[2]  B. de Gelder,et al.  Body expressions influence recognition of emotions in the face and voice. , 2007, Emotion.

[3]  R. Birdwhistell.  Kinesics and Context: Essays on Body Motion Communication , 1971 .

[4]  Jiangtao Wang,et al.  Energy Saving Techniques in Mobile Crowd Sensing: Current State and Future Opportunities , 2018, IEEE Communications Magazine.

[5]  Francisco Herrera,et al.  Cognitive Computing: Architecture, Technologies and Intelligent Applications , 2018, IEEE Access.

[6]  Gregory D. Abowd,et al.  A practical approach for recognizing eating moments with wrist-mounted inertial sensing , 2015, UbiComp.

[7]  Andrea Kleinsmith,et al.  Cross-cultural differences in recognizing affect from body posture , 2006, Interact. Comput..

[8]  Paul Geladi,et al.  Principal Component Analysis , 1987, Comprehensive Chemometrics.

[9]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[10]  Yong Zhu.  Design and Validation of a Low-Cost Portable Device to Quantify Postural Stability † , 2017, Sensors.

[11]  Nasser Kehtarnavaz,et al.  A survey of depth and inertial sensor fusion for human action recognition , 2015, Multimedia Tools and Applications.

[12]  Roger Gassert,et al.  A novel algorithm for detecting active propulsion in wheelchair users following spinal cord injury. , 2016, Medical engineering & physics.

[13]  Kai Tang,et al.  Kernel fusion based extreme learning machine for cross-location activity recognition , 2017, Inf. Fusion.

[14]  H. Wallbott.  Bodily expression of emotion , 1998 .

[15]  Giancarlo Fortino,et al.  Activity recognition and monitoring for smart wheelchair users , 2016, 2016 IEEE 20th International Conference on Computer Supported Cooperative Work in Design (CSCWD).

[16]  Min Chen,et al.  Cognitive Internet of Vehicles , 2018, Comput. Commun..

[17]  Adrian Burns,et al.  SHIMMER™ – A Wireless Sensor Platform for Noninvasive Biomedical Research , 2010, IEEE Sensors Journal.

[18]  Tatsuya Shibata,et al.  Emotion recognition modeling of sitting postures by using pressure sensors and accelerometers , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[19]  Barbara Caputo,et al.  Recognizing human actions: a local SVM approach , 2004, ICPR 2004.

[20]  Lei Wu,et al.  A Fatigue Detect System Based on Activity Recognition , 2014, IDCS.

[21]  Giancarlo Fortino,et al.  Activity Level Assessment Using a Smart Cushion for People with a Sedentary Lifestyle , 2017, Sensors.

[22]  Giancarlo Fortino,et al.  Cloud-Based Wheelchair Assist System for Mobility Impaired Individuals , 2016, IDCS.

[23]  Sajal K. Das,et al.  Care-Chair: Sedentary Activities and Behavior Assessment with Smart Sensing on Chair Backrest , 2016, 2016 IEEE International Conference on Smart Computing (SMARTCOMP).

[24]  Min Chen,et al.  SPHA: Smart Personal Health Advisor Based on Deep Analytics , 2018, IEEE Communications Magazine.

[25]  Ahmad Almogren,et al.  A robust human activity recognition system using smartphone sensors and deep learning , 2018, Future Gener. Comput. Syst..

[26]  Vera Stara,et al.  A Wireless Sensor Insole to Collect and Analyse Gait Data in Real Environment: The WIISEL Project , 2015 .

[27]  Zoubin Ghahramani,et al.  An Introduction to Hidden Markov Models and Bayesian Networks , 2001, Int. J. Pattern Recognit. Artif. Intell..

[28]  Min Chen,et al.  Emotion Communication System , 2017, IEEE Access.

[29]  Yu Zhang,et al.  Activity recognition of wheelchair users based on sequence feature in time-series , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[30]  M. D. Meijer.  The contribution of general features of body movement to the attribution of emotions , 1989 .

[31]  Claire L. Roether,et al.  Critical features for the perception of emotion from gait. , 2009, Journal of vision.

[32]  Roozbeh Jafari,et al.  MotionSynthesis toolset (MoST): a toolset for human motion data synthesis and validation , 2014, MobileHealth '14.

[33]  Andres Upegui,et al.  A Wireless Sensor-Based System for Self-tracking Activity Levels Among Manual Wheelchair Users , 2017, eHealth 360°.

[34]  Zhelong Wang,et al.  Badminton Stroke Recognition Based on Body Sensor Networks , 2016, IEEE Transactions on Human-Machine Systems.

[35]  Kamran Ahsan,et al.  Application specific study, analysis and classification of body area wireless sensor network applications , 2015, Comput. Networks.

[36]  K. Scherer,et al.  Emotion expression in body action and posture. , 2012, Emotion.

[37]  Zhiming Lin,et al.  Large‐Scale and Washable Smart Textiles Based on Triboelectric Nanogenerator Arrays for Self‐Powered Sleeping Monitoring , 2018 .

[38]  Min Chen,et al.  Data-Driven Computing and Caching in 5G Networks: Architecture and Delay Analysis , 2018, IEEE Wireless Communications.

[39]  Paul Lukowicz,et al.  Smart-surface: Large scale textile pressure sensors arrays for activity recognition , 2016, Pervasive Mob. Comput..

[40]  Giancarlo Fortino,et al.  A Task-Oriented Framework for Networked Wearable Computing , 2016, IEEE Transactions on Automation Science and Engineering.

[41]  Athanasios V. Vasilakos,et al.  Body Area Networks: A Survey , 2010, Mob. Networks Appl..

[42]  Irina Rish,et al.  An empirical study of the naive Bayes classifier , 2001 .

[43]  R. Plutchik,et al.  Theories of emotion , 1980 .

[44]  Neeraj Bhargava,et al.  Decision Tree Analysis on J48 Algorithm for Data Mining , 2013 .