Retexturing in the Presence of Complex Illumination and Occlusions

We present a nonrigid registration technique that achieves spatial, photometric, and visibility accuracy. It lets us photo-realistically augment 3D deformable surfaces under complex illumination conditions and in spite of severe occlusions. There are many approaches that address some of these issues but very few that simultaneously handle all of them as we do. We use triangulated meshes to model the geometry and introduce explicit visibility maps as well as separate illumination parameters for each mesh vertex. We cast our registration problem in an expectation maximization framework that allows robust and fully automated operation. It provides explicit illumination and occlusion models that can be used for rendering purposes.

[1]  David A. Forsyth,et al.  Combining Cues: Shape from Shading and Texture , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[2]  David A. Forsyth,et al.  Retexturing Single Views Using Texture and Shading , 2006, ECCV.

[3]  D. Bradley,et al.  Augmenting Non-Rigid Objects with Realistic Lighting , 2004 .

[4]  Sami Romdhani,et al.  Efficient, robust and accurate fitting of a 3D morphable model , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[5]  Luc Van Gool,et al.  A Mean Field EM-algorithm for Coherent Occlusion Handling in MAP-Estimation Prob , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[6]  Ralph Gross,et al.  Active appearance models with occlusion , 2006, Image Vis. Comput..

[7]  Yanxi Liu,et al.  Tracking Dynamic Near-Regular Texture Under Occlusion and Rapid Movements , 2006, ECCV.

[8]  Vincent Lepetit,et al.  Fast Non-Rigid Surface Detection, Registration and Realistic Augmentation , 2008, International Journal of Computer Vision.

[9]  Michel Dhome,et al.  A simple and efficient template matching algorithm , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[10]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[11]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Adrien Bartoli,et al.  Direct Estimation of Non-Rigid Registration , 2004, BMVC.

[13]  Pascal Fua,et al.  Surface Deformation Models for Nonrigid 3D Shape Recovery , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Stan Sclaroff,et al.  Active blobs: region-based, deformable appearance models , 2003, Computer Vision and Image Understanding.

[15]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[16]  Wolfgang Heidrich,et al.  Cloth Motion Capture , 2003, Comput. Graph. Forum.

[17]  W. Heidrich,et al.  Texture Replacement of Garments in Monocular Video Sequences , 2022 .

[18]  Vincent Lepetit,et al.  Augmenting deformable objects in real-time , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[19]  Pascal Fua,et al.  Physically Valid Shape Parameterization for Monocular 3-D Deformable Surface Tracking , 2005, BMVC.