IoMT Based Facial Emotion Recognition System Using Deep Convolution Neural Networks

Facial emotion recognition (FER) is the procedure of identifying human emotions from facial expressions. It is often difficult to identify the stress and anxiety levels of an individual through the visuals captured from computer vision. However, the technology enhancements on the Internet of Medical Things (IoMT) have yielded impressive results from gathering various forms of emotional and physical health-related data. The novel deep learning (DL) algorithms are allowing to perform application in a resource-constrained edge environment, encouraging data from IoMT devices to be processed locally at the edge. This article presents an IoMT based facial emotion detection and recognition system that has been implemented in real-time by utilizing a small, powerful, and resource-constrained device known as Raspberry-Pi with the assistance of deep convolution neural networks. For this purpose, we have conducted one empirical study on the facial emotions of human beings along with the emotional state of human beings using physiological sensors. It then proposes a model for the detection of emotions in real-time on a resource-constrained device, i.e., Raspberry-Pi, along with a co-processor, i.e., Intel Movidius NCS2. The facial emotion detection test accuracy ranged from 56% to 73% using various models, and the accuracy has become 73% performed very well with the FER 2013 dataset in comparison to the state of art results mentioned as 64% maximum. A t-test is performed for extracting the significant difference in systolic, diastolic blood pressure, and the heart rate of an individual watching three different subjects (angry, happy, and neutral).

[1]  Vinod Chandran,et al.  Facial Expression Analysis under Partial Occlusion , 2018, ACM Comput. Surv..

[2]  Alan C. Bovik,et al.  Making long-wave infrared face recognition robust against image quality degradations , 2019, Quantitative InfraRed Thermography Journal.

[3]  Marios Savvides,et al.  CMS-RCNN: Contextual Multi-Scale Region-based CNN for Unconstrained Face Detection , 2016, ArXiv.

[4]  Wahida Handouzi,et al.  Facial emotion recognition using deep learning: review and insights , 2020, FNC/MobiSPC.

[5]  Vinayak B. Kulkarni,et al.  Face Recognition Using Golden Ratio for Door Access Control System , 2021 .

[6]  Torki A. Altameem,et al.  Facial expression recognition using human machine interaction and multi-modal visualization analysis for healthcare applications , 2020, Image Vis. Comput..

[7]  Fei Yan,et al.  Real-time facial emotion recognition using lightweight convolution neural network , 2021 .

[8]  Sridha Sridharan,et al.  Using Synthetic Data to Improve Facial Expression Analysis with 3D Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[9]  Yang Xiao,et al.  Energy-efficient node scheduling algorithms for wireless sensor networks using Markov Random Field model , 2016, Inf. Sci..

[10]  Yanjiao Chen,et al.  WiFace: Facial Expression Recognition Using Wi-Fi Signals , 2022, IEEE Trans. Mob. Comput..

[11]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[12]  Mark Elshaw,et al.  A hybrid deep learning neural approach for emotion recognition from facial expressions for socially assistive robots , 2018, Neural Computing and Applications.

[13]  Yu-Sheng Su,et al.  Predicting behavioral competencies automatically from facial expressions in real-time video-recorded interviews , 2021, Journal of Real-Time Image Processing.

[14]  Md. Zia Uddin,et al.  Emotion recognition using speech and neural structured learning to facilitate edge intelligence , 2020, Eng. Appl. Artif. Intell..

[16]  Ioannis Pitas,et al.  Facial expression analysis under partial occlusion , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[17]  Naoyuki Kubota,et al.  Attention mechanism-based CNN for facial expression recognition , 2020, Neurocomputing.

[18]  Sung Wook Baik,et al.  Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services , 2019, Inf. Sci..

[19]  Burhan Ergen,et al.  Facial emotion recognition on a dataset using convolutional neural network , 2017, 2017 International Artificial Intelligence and Data Processing Symposium (IDAP).

[20]  Fahad Layth Malallah,et al.  Controlling Embedded Systems Remotely via Internet-of-Things Based on Emotional Recognition , 2020, Adv. Hum. Comput. Interact..

[21]  M. Shamim Hossain,et al.  Deep learning-based intelligent face recognition in IoT-cloud environment , 2020, Comput. Commun..

[22]  Yang Yang,et al.  Cross-domain facial expression recognition via an intra-category common feature and inter-category Distinction feature fusion network , 2019, Neurocomputing.

[23]  M. A. Rahman,et al.  An Internet-of-Medical-Things-Enabled Edge Computing Framework for Tackling COVID-19 , 2021, IEEE Internet of Things Journal.

[25]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  K. P. Sridhar,et al.  LAMSTAR: For IoT‐based face recognition system to manage the safety factor in smart cities , 2019, Trans. Emerg. Telecommun. Technol..

[27]  Hung-Hsu Tsai,et al.  Facial expression recognition using a combination of multiple facial features and support vector machine , 2018, Soft Comput..

[28]  Wenzhong Guo,et al.  Robust co-clustering via dual local learning and high-order matrix factorization , 2017, Knowl. Based Syst..

[29]  Hamid Sadeghi,et al.  Human vision inspired feature extraction for facial expression recognition , 2019, Multimedia Tools and Applications.

[30]  Anand Nayyar,et al.  BioSenHealth 1.0: A Novel Internet of Medical Things (IoMT)-Based Patient Health Monitoring System , 2018, International Conference on Innovative Computing and Communications.

[31]  Naim Ahmad,et al.  Internet of medical things: Architectural model, motivational factors and impediments , 2018, 2018 15th Learning and Technology Conference (L&T).

[32]  Michael Goh Kah Ong,et al.  Facial Expression Recognition Using a Hybrid CNN-SIFT Aggregator , 2017, MIWAI.

[33]  Chris Yakopcic,et al.  A State-of-the-Art Survey on Deep Learning Theory and Architectures , 2019, Electronics.

[34]  Sung Wook Baik,et al.  Facial appearance and texture feature-based robust facial expression recognition framework for sentiment knowledge discovery , 2017, Cluster Computing.

[35]  Felan Carlo C. Garcia,et al.  Emotion Recognition via Facial Expression: Utilization of Numerous Feature Descriptors in Different Machine Learning Algorithms , 2018, TENCON 2018 - 2018 IEEE Region 10 Conference.

[36]  Najla Al-Nabhan,et al.  Emotion-Aware and Intelligent Internet of Medical Things Toward Emotion Recognition During COVID-19 Pandemic , 2021, IEEE Internet of Things Journal.

[37]  Ron Kimmel,et al.  A Deep Learning Perspective on the Origin of Facial Expressions , 2017, ArXiv.

[38]  Ninad Mehendale,et al.  Facial emotion recognition using convolutional neural networks (FERC) , 2020, SN Applied Sciences.

[39]  Gaurav Dhiman,et al.  An Innovative Approach for Face Recognition Using Raspberry Pi , 2020, Artificial Intelligence Evolution.

[40]  M. Shamim Hossain,et al.  Privacy-Enhanced Data Fusion for COVID-19 Applications in Intelligent Internet of Medical Things , 2020, IEEE Internet of Things Journal.

[41]  Geng Yang,et al.  cGAN Based Facial Expression Recognition for Human-Robot Interaction , 2019, IEEE Access.

[42]  António J. R. Neves,et al.  Facial Expression Recognition Using Computer Vision: A Systematic Review , 2019, Applied Sciences.

[43]  Fasih Haider,et al.  Emotion Recognition in Low-Resource Settings: An Evaluation of Automatic Feature Selection Methods , 2019, Comput. Speech Lang..