Deep learning based facial expressions recognition system for assisting visually impaired persons

In general, a good interaction including communication can be achieved when verbal and non-verbal information such as body movements, gestures, facial expressions, can be processed in two directions between the speaker and listener. Especially the facial expression is one of the indicators of the inner state of the speaker and/or the listener during the communication. Therefore, recognizing the facial expressions is necessary and becomes the important ability in communication. Such ability will be a challenge for the visually impaired persons. This fact motivated us to develop a facial recognition system. Our system is based on deep learning algorithm. We implemented the proposed system on a wearable device which enables the visually impaired persons to recognize facial expressions during the communication. We have conducted several experiments involving the visually impaired persons to validate our proposed system and the promising results were achieved.

[1]  Tomoaki Nakamura,et al.  Visual Recognition System for Cleaning Tasks by Humanoid Robots , 2013 .

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Michael J. Lyons,et al.  Evidence and a computational explanation of cultural differences in facial expression recognition. , 2010, Emotion.

[6]  Hendrik P. Buimer,et al.  Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device , 2018, PloS one.

[7]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[8]  Charbel Sakr,et al.  Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm , 2018, ICLR.

[9]  Djoko Purwanto,et al.  Particle Filter with Integrated Multiple Features for Object Detection and Tracking , 2018 .

[10]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[11]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Tomoaki Nakamura,et al.  Hierarchical multilevel object recognition using Markov model , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[13]  Nuno Vasconcelos,et al.  Cascade R-CNN: Delving Into High Quality Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Yuhang Zhao,et al.  A Face Recognition Application for People with Visual Impairments: Understanding Use Beyond the Lab , 2018, CHI.

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Tomoaki Nakamura,et al.  Real-time 3D visual sensor for robust object recognition , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[18]  Bernard Chazelle,et al.  Shape distributions , 2002, TOGS.

[19]  Minho Lee,et al.  Smart Cane: Face Recognition System for Blind , 2015, HAI.

[20]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Shifeng Zhang,et al.  Single-Shot Refinement Neural Network for Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Tomoaki Nakamura,et al.  Modeling of honest signals for human robot interaction , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[23]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[24]  Ronny Mardiyanto,et al.  Inclined Image Recognition for Aerial Mapping using Deep Learning and Tree based Models , 2018 .

[25]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[26]  Larry S. Davis,et al.  Soft-NMS — Improving Object Detection with One Line of Code , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Andrea Vedaldi,et al.  Weakly Supervised Deep Detection Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Kai Chen,et al.  Hybrid Task Cascade for Instance Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Takashi Omori,et al.  Attention Estimation for Child-Robot Interaction , 2016, HAI.

[31]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[33]  Takayuki Nagai,et al.  Toward playmate robots that can play with children considering personality , 2014, HAI.

[34]  Stephen Lin,et al.  Deformable ConvNets V2: More Deformable, Better Results , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  P. Lindsay,et al.  Social inclusion of employees who are blind or low vision , 2011 .