Estimating head pose and state of facial elements for sign language video

Currently there is an increasing need of automatic video analysis and annotation tools to support linguists in their studies of sign language. Up to now, the amount of studies focusing on automatic annotation of non-manual gestures in sign language videos have been limited (e.g. Metaxas et al., 2012). Therefore, in this work we study methods for automatic estimation of three head pose angles and the state of facial elements (eyebrow position, eye openness, and mouth state). The estimation methods we propose are incorporated in our publicly available SLMotion software package (Karppa et al., 2014) for sign language video processing and analysis. Head pose can be described by the three angles: yaw, pitch, and roll (Figure 1a). We propose an approach for head pose estimation from images based on two kinds of visual features. The first group of features is formed by the set of facial landmark points extracted using the flandmark software library (Uřicař et al., 2012). Secondly, as novel additional features we use a tonal segmentation mask of skin-like colors within the face bounding box. The roll angle is estimated using a geometric approach based on the location of the eye landmarks. The yaw and pitch angles are estimated with separate Support Vector Regression (Smola and Scholkopf, 2004) classifiers. The classifiers are trained using the full set of features extracted from a subset of 684 annotated images from the Pointing04 image database (Gourier et al., 2004) of subjects performing various head poses. In the image subset, the pose varies from −45◦ to +45◦ in yaw and from −30◦ to +30◦ in pitch, with 15◦ steps from one pose to another. The performance of the proposed head pose estimation method was evaluated in two experimental settings. In the first series of experiments a separate subset of the Pointing04 data was used to measure the accuracy of the trained yaw and pitch regressors. In these experiments, our accuracy levels compare favorably with published results on the same data. Some of the observed advantage might be due to the use of only near frontal pose angles in our evaluation. Roll detection was not evaluated as the Pointing04 database does not include reference annotations for roll. In the second experiment continuous signing was recorded both by a video camera and motion capture equipment (Figure 2). The pose angles were estimated from video frames and the estimates compared with the ground truth values from the motion capture recording. In this experiment the estimated angles for yaw and roll had a very high correlation with the ground truth, whereas for pitch the correlation was slightly lower. We conclude that our video-based estimators are very well comparable with motion capture measurements. The method we propose for estimating eyebrow position, eye openness, and mouth state is based on the construction of an extended set of facial landmarks (Figure 1b). These estimate the position of eyebrows, eyelids, and upper and lower lip boundaries which are not part of the flandmark output. The detection of extended landmarks is initialized by using the flandmark set for coarse location identification of facial elements. The proposed landmark detection method then employs an ensemble of techniques for each facial element: oriented projections and pixel similarity for eyebrows, oriented projections and radial transform for eyes, and pseudo-hue mask for lips. We also consider landmarks detected using the recently presented appearance-based Supervised Descent Method that is implemented in the IntraFace software package (Human Sensing Laboratory and Affect Analysis Group, 2013). The extended landmarks are used to calculate a set of geometric features which are further postprocessed with Principal Component Analysis. The processed features function as input to statistical classifiers trained to produce quantized estimates of eyebrow position, eye openness, and mouth state (Table 1). We evaluated the performance of our facial element state estimators in a quantitative and qualitative type of experiments. For the first experiment set we manually annotated the facial states in videos taken from the SUVI dictionary of Finnish Sign Language (Suvi, 2003). The annotations were performed frame-by-frame on basis of the visual appearance of the isolated frame, without regard to linguistic significance of the face states. We evaluated our automatic estimators against the manual annotations using the Mathews’ Correlation Coefficient (Powers, 2011). In the experiments (Figure 3), eye openness and vertical mouth state estimators had high correlations with the ground truth. The eyebrow position and horizontal mouth state estimators were quite noisy, often missing subtle changes and being sensitive to small variations in head pose. For the qualitative set of experiments we employed annotations prepared for a subset of the SUVI material in an earlier work (Jantunen, 2007) from the point of view of linguistic significance. In the evaluation example (Figure 3), the automatic estimations correctly detected blinks and squints with few mislabeled frames. Furthermore, the eyebrow estimations coincided with the linguistic annotations except in the non-linguistic visual changes or perspective illusions (head tilting). In summary, we have proposed methods for automatic estimation of two types of non-manual elements from sign language video: head pose and facial element state. The experimental results testify of a promising progress on these two separate fronts towards automatic annotation of non-manuals in sign language.