Regularized Bundle-Adjustment to Model Heads from Image Sequences without Calibration Data

We address the structure-from-motion problem in the context of head modeling from video sequences for which calibration data is not available. This task is made challenging by the fact that correspondences are difficult to establish due to lack of texture and that a quasi-euclidean representation is required for realism.We have developed an approach based on regularized bundle-adjustment. It takes advantage of our rough knowledge of the head's shape, in the form of a generic face model. It allows us to recover relative head-motion and epipolar geometry accurately and consistently enough to exploit a previously-developed stereo-based approach to head modeling. In this way, complete and realistic head models can be acquired with a cheap and entirely passive sensor, such as an ordinary video camera, with minimal manual intervention.We chose to demonstrate and evaluate our technique mainly in the context of head-modeling. We do so because it is the application for which all the tools required to perform the complete reconstruction are available to us. We will, however, argue that the approach is generic and could be applied to other tasks, such as body modeling, for which generic facetized models exist.

[1]  Rajiv Gupta,et al.  Stereo from uncalibrated cameras , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Pascal Fua,et al.  A parallel stereo algorithm that produces dense depth maps and preserves image features , 1993, Machine Vision and Applications.

[3]  Timothy F. Cootes,et al.  A unified approach to coding and interpreting face images , 1995, Proceedings of IEEE International Conference on Computer Vision.

[4]  Reinhard Koch,et al.  Self-Calibration and Metric Reconstruction Inspite of Varying and Unknown Intrinsic Camera Parameters , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[5]  Peter F. Sturm,et al.  Critical motion sequences for monocular self-calibration and uncalibrated Euclidean reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Bill Triggs,et al.  Autocalibration and the absolute quadric , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Yiannis Aloimonos,et al.  Changes in Surface Convexity and Topology Caused by Distortions of Stereoscopic Visual Space , 1998, ECCV.

[8]  Dimitris N. Metaxas,et al.  Deformable model-based shape and motion analysis from images using motion residual error , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[9]  Demetri Terzopoulos,et al.  Realistic modeling for facial animation , 1995, SIGGRAPH.

[10]  Alex Pentland,et al.  Parametrized structure from motion for 3D adaptive feedback tracking of faces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[11]  William H. Press,et al.  Numerical recipes , 1990 .

[12]  Thierry Viéville,et al.  Canonical Representations for the Geometries of Multiple Projective Views , 1996, Comput. Vis. Image Underst..

[13]  J. Z. Zhu,et al.  The finite element method , 1977 .

[14]  Thomas S. Huang,et al.  BOOK REVIEW: Calibration and Orientation of Cameras in Computer Vision , 2001 .

[15]  Dimitris N. Metaxas,et al.  Incorporating illumination constraints in deformable models , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[16]  Pascal Fua,et al.  From Regular Images to Animated Heads: A Least Squares Approach , 1998, ECCV.

[17]  J. Tukey,et al.  The Fitting of Power Series, Meaning Polynomials, Illustrated on Band-Spectroscopic Data , 1974 .

[18]  Horst A. Beyer,et al.  System Calibration Through Self-Calibration , 2001 .

[19]  Pascal Fua,et al.  Animated Heads from Ordinary Images: A Least-Squares Approach , 1999, Comput. Vis. Image Underst..

[20]  Jean-Philippe Tarel,et al.  Camcal v1.0 Manual A Complete Software Solution for Camera Calibration , 1996 .

[21]  Andrew Zisserman,et al.  Resolving ambiguities in auto–calibration , 1998, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[22]  Thomas Vetter,et al.  A morphable model for the synthesis of 3D faces , 1999, SIGGRAPH.

[23]  Sing Bing Kang.  A Structure from Motion Approach using Constrained Deformable Models and Appearance Prediction , 2002 .

[24]  Emiko Kudo.  Hawaii , 1982 .

[25]  Anna Kochan.  A united approach , 1991 .

[26]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[27]  W. Press,et al.  Numerical Recipes: The Art of Scientific Computing , 1987 .

[28]  Thomas S. Huang,et al.  Analysis-based facial expression synthesis , 1994, Proceedings of 1st International Conference on Image Processing.

[29]  Daniel Thalmann,et al.  Fast realistic human body deformations for animation and VR applications , 1996, Proceedings of CG International '96.

[30]  Luc Van Gool,et al.  Active acquisition of 3D shape for moving objects , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[31]  Olivier D. Faugeras,et al.  Computing differential properties of 3-D shapes from stereoscopic images without 3-D models , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Daniel Thalmann,et al.  The HUMANOID Environment for Interactive Animation of Multiple Deformable Human Characters , 1995, Comput. Graph. Forum.

[33]  Daniel Thalmann,et al.  Simulation of Facial Muscle Actions Based on Rational Free Form Deformations , 1992, Comput. Graph. Forum.

[34]  Aaron F. Bobick,et al.  The direct computation of height from shading , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  D. Gennery,et al.  Calibration and Orientation of Cameras in Computer Vision , 2001 .

[36]  WON-SOOK LEE,et al.  From Real Faces To Virtual Faces : Problems and Solutions , 1999 .

[37]  Andrew W. Fitzgibbon,et al.  Automatic Camera Recovery for Closed or Open Image Sequences , 1998, ECCV.

[38]  Pascal Fua,et al.  Object-centered surface reconstruction: Combining multi-image stereo and shading , 1995, International Journal of Computer Vision.

[39]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[40]  Paul A. Beardsley,et al.  Sequential Updating of Projective and Affine Structure from Motion , 1997, International Journal of Computer Vision.

[41]  Volume 26 , 2002 .

[42]  O. D. Faugeras,et al.  Camera Self-Calibration: Theory and Experiments , 1992, ECCV.