All-in-one “HairNet”: A Deep Neural Model for Joint Hair Segmentation and Characterization

The hair appearance is among the most valuable soft biometric traits when performing human recognition at-a-distance. Even in degraded data, the hair's appearance is instinctively used by humans to distinguish between individuals. In this paper we propose a multi-task deep neural model capable of segmenting the hair region, while also inferring the hair color, shape and style, all from in-the-wild images. Our main contributions are two-fold: 1) the design of an all-in-one neural network, based on depthwise separable convolutions to extract the features; and 2) the use convolutional feature masking layer as an attention mechanism that enforces the analysis only within the ‘hair’ regions. In a conceptual perspective, the strength of our model is that the segmentation mask is used by the other tasks to perceive - at feature-map level - only the regions relevant to the attribute characterization task. This paradigm allows the network to analyze features from nonrectangular areas of the input data, which is particularly important, considering the irregularity of hair regions. Our experiments showed that the proposed approach reaches a hair segmentation performance comparable to the state-of-the-art, having as main advantage the fact of performing multiple levels of analysis in a single-shot paradigm.

[1]  Jian Sun,et al.  Convolutional feature masking for joint object and stuff segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Wei-Yi Chang,et al.  FATAUVA-Net: An Integrated Deep Learning Framework for Facial Attribute Recognition, Action Unit Detection, and Valence-Arousal Estimation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Carlos D. Castillo,et al.  An All-In-One Convolutional Neural Network for Face Analysis , 2016, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[5]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[6]  Erik G. Learned-Miller,et al.  Unsupervised Joint Alignment of Complex Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[7]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Julian Fiérrez,et al.  Soft Biometrics and Their Application in Person Recognition at a Distance , 2014, IEEE Transactions on Information Forensics and Security.

[9]  Dan Wang,et al.  Isomorphic Manifold Inference for hair segmentation , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[10]  Dragomir Anguelov,et al.  Markov random field models for hair and face segmentation , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[11]  U. Lipowezky,et al.  Using integrated color and texture features for automatic hair detection , 2008, 2008 IEEE 25th Convention of Electrical and Electronics Engineers in Israel.

[12]  Jian Sun,et al.  BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[13]  Pawan Sinha,et al.  Face Recognition by Humans: Nineteen Results All Computer Vision Researchers Should Know About , 2006, Proceedings of the IEEE.

[14]  Adrian Sergiu Darabant,et al.  Fast In-the-Wild Hair Segmentation and Color Classification , 2019, VISIGRAPP.

[15]  Adrien Bartoli,et al.  Automatic Hair Detection in the Wild , 2010, 2010 20th International Conference on Pattern Recognition.

[16]  Eam Khwang Teoh,et al.  Human Hair Segmentation and Length Detection for Human Appearance Model , 2014, 2014 22nd International Conference on Pattern Recognition.

[17]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[18]  Zhenyun Peng,et al.  Image Based Hair Segmentation Algorithm for the Application of Automatic Facial Caricature Synthesis , 2014, TheScientificWorldJournal.

[19]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[20]  A. Hurlbert,et al.  Understanding colour perception and preference , 2012 .

[21]  Riccardo Leonardi,et al.  Figaro, hair detection and segmentation in the wild , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[22]  Pierre-Yves Coulon,et al.  Frequential and color analysis for hair mask segmentation , 2008, 2008 15th IEEE International Conference on Image Processing.

[23]  Larry S. Davis,et al.  Detection and analysis of hair , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Seongdo Kim,et al.  Automatic skin and hair masking using fully convolutional networks , 2017, 2017 IEEE International Conference on Multimedia and Expo (ICME).

[25]  John Stobo,et al.  United We Stand , 1993, Annals of Internal Medicine.

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[28]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[30]  Malay Kishore Dutta,et al.  Hair Segmentation for Color Estimation in Surveillance Systems , 2014, MMEDIA 2014.

[31]  Adrian Sergiu Darabant,et al.  A Deep Learning Approach to Hair Segmentation and Color Extraction from Facial Images , 2018, ACIVS.

[32]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[34]  Hugo Proença,et al.  Soft Biometrics: Globally Coherent Solutions for Hair Segmentation and Style Recognition Based on Hierarchical MRFs , 2017, IEEE Transactions on Information Forensics and Security.

[35]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[36]  Parham Aarabi,et al.  Real-Time Deep Hair Matting on Mobile Devices , 2017, 2018 15th Conference on Computer and Robot Vision (CRV).

[37]  P Sinha,et al.  Last but Not Least , 2000, Perception.