Parametrized structure from motion for 3D adaptive feedback tracking of faces

A real-time system is described for automatically detecting, modeling and tracking faces in 3D. A closed loop approach is proposed which utilizes structure from motion to generate a 3D model of a face and then feed back the estimated structure to constrain feature tracking in the next frame. The system initializes by using skin classification, symmetry operations, 3D warping and eigenfaces to find a face. Feature trajectories are then computed by SSD or correlation-based tracking. The trajectories are simultaneously processed by an extended Kalman filter to stably recover 3D structure, camera geometry and facial pose. Adaptively weighted estimation is used in this filter by modeling the noise characteristics of the 2D image patch tracking technique. In addition, the structural estimate is constrained by using parametrized models of facial structure (eigen-heads). The Kalman filter's estimate of the 3D state and motion of the face predicts the trajectory of the features which constrains the search space for the next frame in the video sequence. The feature tracking and Kalman filtering closed loop system operates at 25 Hz.

[1]  Yehezkel Yeshurun,et al.  Robust detection of facial features by generalized symmetry , 1992, [1992] Proceedings. 11th IAPR International Conference on Pattern Recognition.

[2]  Martin D. Levine,et al.  Annular symmetry operators: a method for locating and describing objects , 1995, Proceedings of IEEE International Conference on Computer Vision.

[3]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[4]  Alex Waibel,et al.  Gaze Tracking Based on Face‐Color , 1995 .

[5]  Alex Pentland,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Gregory D. Hager,et al.  Real-time tracking of image regions with changes in geometry and illumination , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  P PentlandAlex,et al.  Recursive Estimation of Motion, Structure, and Focal Length , 1995 .

[8]  Alex Pentland,et al.  Probabilistic visual learning for object detection , 1995, Proceedings of IEEE International Conference on Computer Vision.

[9]  Tony Jebara,et al.  3D Pose Estimation and Normalization for Face Recognition , 1995 .

[10]  T. D. Alter.  D Pose from 3 Corresponding Points under Weak-Perspective Projection , 1992 .

[11]  G. Sela,et al.  Fast computation of multiscalar symmetry in foveated images , 1995, Proceedings of Conference on Computer Architectures for Machine Perception.