Deep learning-based facial expression recognition for monitoring neurological disorders

Facial expressions play an important role in communication. Impaired facial expression is a common sign of numerous medical conditions, particularly neurological disorders. Accurate automated systems are needed to recognize facial expressions and to reveal valuable information that can be used for diagnosis and monitoring of neurological disorders. This paper presents a novel deep learning approach for automatic facial expression recognition. The proposed architecture first segments the facial components known to be important for facial expression recognition and forms an iconized image; then performs facial expression classification using the obtained iconized facial components image combined with the raw facial images. This approach integrates local part-based features with holistic facial information for robust facial expression recognition. Preliminary experimental results using the proposed system achieved 93.43% facial expression recognition accuracy, more than 6% accuracy improvement compared to facial expression recognition from raw input images. The goal of the proposed study is design of a noninvasive, objective, and quantitative facial expression recognition system to assist diagnosis and monitoring of neurological disorders affecting facial expressions.

[1]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Qingshan Liu,et al.  Learning active facial patches for expression analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  F. Bunyak,et al.  FACIAL COMPONENT SEGMENTATION USING CONVOLUTIONAL NEURAL NETWORK , 2018 .

[5]  Radhika M. Pai,et al.  Automatic Facial Expression Recognition Using DCNN , 2016 .

[6]  Khan M. Iftekharuddin,et al.  Deep learning of texture and structural features for multiclass Alzheimer's disease classification , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[7]  Yun-Su Chung,et al.  Deep Learning-Based Vehicle Classification Using an Ensemble of Local Expert and Global Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[9]  Lisa L Hunter,et al.  Clinical Practice Guideline , 2016, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[10]  J. Kilner,et al.  Facial Emotion Recognition and Expression in Parkinson’s Disease: An Emotional Mirror Mechanism? , 2017, PloS one.

[11]  Sung-Jea Ko,et al.  Illumination normalisation using convolutional neural network with application to face recognition , 2017 .

[12]  Albert Ali Salah,et al.  Kernel ELM and CNN Based Facial Age Estimation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  Frank A. Russo,et al.  Deficits in the Mimicry of Facial Expressions in Parkinson's Disease , 2016, Front. Psychol..

[14]  Hongmei Wen,et al.  Weakness of Eye Closure with Central Facial Paralysis after Unilateral Hemispheric Stroke Predicts a Worse Outcome. , 2017, Journal of stroke and cerebrovascular diseases : the official journal of National Stroke Association.

[15]  K. Sundaraj,et al.  Methods and approaches on emotions recognition in neurodegenerative disorders: A review , 2012, 2012 IEEE Symposium on Industrial Electronics and Applications.

[16]  Jiwen Lu,et al.  Ordinal Deep Feature Learning for Facial Age Estimation , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[17]  Geoffrey Bird,et al.  Can Neurotypical Individuals Read Autistic Facial Expressions? Atypical Production of Emotional Facial Expressions in Autism Spectrum Disorders , 2015, Autism research : official journal of the International Society for Autism Research.

[18]  Chris Oliver,et al.  Brief Report: A Longitudinal Study of Excessive Smiling and Laughing in Children with Angelman Syndrome , 2015, Journal of autism and developmental disorders.

[19]  Matthias Scheutz,et al.  Preserving dignity in patient caregiver relationships using moral emotions and robots , 2014, 2014 IEEE International Symposium on Ethics in Science, Technology and Engineering.

[20]  B. Miller,et al.  A neural network underlying intentional emotional facial expression in neurodegenerative disease , 2017, NeuroImage: Clinical.

[21]  Jiasong Zhu,et al.  Object specific deep feature and its application to face detection , 2017, 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA).

[22]  Vitoantonio Bevilacqua,et al.  A new tool to support diagnosis of neurological disorders by means of facial expressions , 2011, 2011 IEEE International Symposium on Medical Measurements and Applications.

[23]  Sandra E. Black,et al.  Impaired recognition of negative facial emotions in patients with frontotemporal dementia , 2005, Neuropsychologia.

[24]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[25]  Tanaya Guha,et al.  On quantifying facial expression-related atypicality of children with Autism Spectrum Disorder , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[26]  Haifeng Hu,et al.  Facial expression recognition with FRR-CNN , 2017 .

[27]  James L. Netterville,et al.  Clinical Practice Guideline , 2013, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[28]  Xiangji Huang,et al.  CNN-based image analysis for malaria diagnosis , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[29]  Steven Rosenbaum,et al.  Clinical Practice Guideline , 2013, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[30]  Yann LeCun,et al.  Generalization and network design strategies , 1989 .

[31]  L. Lundqvist,et al.  Prevalence of orofacial dysfunction in cerebral palsy and its association with gross motor function and manual ability , 2016, Developmental medicine and child neurology.

[32]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Raquel E Gur,et al.  Emotion-discrimination deficits in mild Alzheimer disease. , 2005, The American journal of geriatric psychiatry : official journal of the American Association for Geriatric Psychiatry.

[34]  Adel Hafiane,et al.  Integrating segmentation with deep learning for enhanced classification of epithelial and stromal tissues in H&E images , 2017, Pattern Recognit. Lett..