Video-rate capture of dynamic face shape and appearance

This paper presents a system for simultaneous capture of video sequences of face shape and colour appearance. Shape capture uses a projected infrared structured light pattern together with stereo reconstruction to simultaneously acquire full resolution shape and colour image sequences at video rate. Displacement mapping techniques are introduced to represent dynamic face surface shape as a displacement video. This unifies the representation of face shape and colour. The displacement video representation enables efficient registration, integration and spatiotemporal analysis of captured face data. Results demonstrate that the system achieves video-rate (25 Hz) acquisition of dynamic 3D colour faces at PAL resolution with an rms accuracy of 0.2 mm and a visual quality comparable to the captured video.

[1]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Pascal Fua,et al.  Object-centered surface reconstruction: Combining multi-image stereo and shading , 1995, International Journal of Computer Vision.

[3]  Zicheng Liu,et al.  Model-based bundle adjustment with application to face modeling , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[4]  Takeo Kanade,et al.  A locally adaptive window for signal matching , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[5]  Pascal Fua,et al.  Regularized Bundle-Adjustment to Model Heads from Image Sequences without Calibration Data , 2000, International Journal of Computer Vision.

[6]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[7]  Luc Van Gool,et al.  Realistic face animation for speech , 2002, Comput. Animat. Virtual Worlds.

[8]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[9]  Shree K. Nayar,et al.  Telecentric Optics for Computational Vision , 1996, ECCV.

[10]  Luc Van Gool,et al.  Active acquisition of 3D shape for moving objects , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[11]  Shree K. Nayar,et al.  Real-time focus range sensor , 1995, Proceedings of IEEE International Conference on Computer Vision.

[12]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[13]  Nadia Magnenat-Thalmann,et al.  Head Modeling from Pictures and Morphing in 3D with Image Metamorphosis Based on Triangulation , 1998, CAPTECH.

[14]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Takeo Kanade,et al.  A locally adaptive window for signal matching , 2004, International Journal of Computer Vision.

[16]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[17]  Adrian Hilton,et al.  Implicit Surface-Based Geometric Fusion , 1998, Comput. Vis. Image Underst..

[18]  Andrew Blake,et al.  Real-time tracking of surfaces with structured light , 1994, Image Vis. Comput..