Visual Diver Face Recognition for Underwater Human-Robot Interaction

This paper presents a deep-learned facial recognition method for underwater robots to identify scuba divers. Specifically, the proposed method is able to recognize divers underwater with faces heavily obscured by scuba masks and breathing apparatus. Our contribution in this research is towards robust facial identification of individuals under significant occlusion of facial features and image degradation from underwater optical distortions. With the ability to correctly recognize divers, autonomous underwater vehicles (AUV) will be able to engage in collaborative tasks with the correct person in human-robot teams and ensure that instructions are accepted from only those authorized to command the robots. We demonstrate that our proposed framework is able to learn discriminative features from real-world diver faces through different data augmentation and generation techniques. Experimental evaluations show that this framework achieves a 3-fold increase in prediction accuracy compared to the state-of-the-art (SOTA) algorithms and is well-suited for embedded inference on robotic platforms.

[1]  Zhenan Sun,et al.  Dynamic Feature Learning for Partial Face Recognition , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[3]  Irene Kotsia,et al.  RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Adrian Galdran,et al.  Automatic Red-Channel underwater image restoration , 2015, J. Vis. Commun. Image Represent..

[6]  Md Jahidul Islam,et al.  Robotic Detection of Marine Litter Using Deep Visual Detection Models , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[7]  Shifeng Zhang,et al.  S^3FD: Single Shot Scale-Invariant Face Detector , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Bhiksha Raj,et al.  SphereFace: Deep Hypersphere Embedding for Face Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Carlos D. Castillo,et al.  Frontal to profile face verification in the wild , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[11]  Gregory Dudek,et al.  A Visual Language for Robot Control and Programming: A Human-Interface Study , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[12]  Shifeng Zhang,et al.  Improved Selective Refinement Network for Face Detection , 2019, ArXiv.

[13]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[14]  Maja Pantic,et al.  A real-time and unsupervised face Re-Identification system for Human-Robot Interaction , 2018, Pattern Recognit. Lett..

[15]  Md Jahidul Islam,et al.  Dynamic Reconfiguration of Mission Parameters in Underwater Human-Robot Collaboration , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Wei Liu,et al.  Occlusion Robust Face Recognition Based on Mask Learning With Pairwise Differential Siamese Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Junaed Sattar,et al.  Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception , 2020, Robotics: Science and Systems.

[18]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[19]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[20]  Michael R. M. Jenkin,et al.  Swimming with robots: Human robot communication at depth , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  Junaed Sattar,et al.  Realtime Multi-Diver Tracking and Re-identification for Underwater Human-Robot Collaboration , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[23]  Jason M. O'Kane,et al.  Navigation in the Presence of Obstacles for an Agile Autonomous Underwater Vehicle , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[25]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Steven C. H. Hoi,et al.  Face Detection using Deep Learning: An Improved Faster RCNN Approach , 2017, Neurocomputing.

[27]  Alexei A. Efros,et al.  Contrastive Learning for Unpaired Image-to-Image Translation , 2020, ECCV.

[28]  Mei Wang,et al.  Deep Face Recognition: A Survey , 2018, Neurocomputing.

[29]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[30]  Ioannis Patras,et al.  Robust Face Alignment Under Occlusion via Regional Predictive Power Estimation , 2015, IEEE Transactions on Image Processing.

[31]  Jung-Woo Ha,et al.  StarGAN v2: Diverse Image Synthesis for Multiple Domains , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[33]  Xu Tang,et al.  PyramidBox: A Context-assisted Single Shot Face Detector , 2018, ECCV.

[34]  Shuicheng Yan,et al.  Robust LSTM-Autoencoders for Face De-Occlusion in the Wild , 2016, IEEE Transactions on Image Processing.

[35]  Hassan Ugail,et al.  Deep face recognition using imperfect facial data , 2019, Future Gener. Comput. Syst..

[36]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Devendra Patil,et al.  Disguised Face Identification (DFI) with Facial KeyPoints Using Spatial Fusion Convolutional Network , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[38]  Alberto Quattrini Li,et al.  Underwater cave mapping using stereo vision , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[39]  Yu Qiao,et al.  A Discriminative Feature Learning Approach for Deep Face Recognition , 2016, ECCV.

[40]  Tal Hassner,et al.  Effective face frontalization in unconstrained images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Ming-Hsuan Yang,et al.  Generative Face Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Jian Yang,et al.  Robust sparse coding for face recognition , 2011, CVPR 2011.

[43]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Li Bai,et al.  Cosine Similarity Metric Learning for Face Verification , 2010, ACCV.

[45]  Qian Zhang,et al.  VarGFaceNet: An Efficient Variable Group Convolutional Neural Network for Lightweight Face Recognition , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[46]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Junaed Sattar,et al.  Visual Diver Recognition for Underwater Human-Robot Collaboration , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[48]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Jian Yang,et al.  DSFD: Dual Shot Face Detector , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Xing Ji,et al.  CosFace: Large Margin Cosine Loss for Deep Face Recognition , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[52]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[53]  Md Jahidul Islam,et al.  Understanding human motion and gestures for underwater human–robot collaboration , 2018, J. Field Robotics.

[54]  Yihong Gong,et al.  Robust Deep Auto-encoder for Occluded Face Recognition , 2015, ACM Multimedia.

[55]  Shuo Yang,et al.  WIDER FACE: A Face Detection Benchmark , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Wei-Chen Chiu,et al.  Learning Face Recognition Unsupervisedly by Disentanglement and Self-Augmentation , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[57]  Carlos D. Castillo,et al.  Triplet probabilistic embedding for face verification and clustering , 2016, 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[58]  Xiaogang Wang,et al.  Deep Learning Face Representation by Joint Identification-Verification , 2014, NIPS.

[59]  E. Schwalbe GEOMETRIC MODELLING AND CALIBRATION OF FISHEYE LENS CAMERA SYSTEMS , 2005 .

[60]  Florian Shkurti,et al.  Vision-Based Goal-Conditioned Policies for Underwater Navigation in the Presence of Obstacles , 2020, RSS 2020.

[61]  Chung-Lin Huang,et al.  Human Object Identification for Human-Robot Interaction by Using Fast R-CNN , 2018, 2018 Second IEEE International Conference on Robotic Computing (IRC).