Illumination-based image synthesis: creating novel images of human faces under differing pose and lighting

We present an illumination-based method for synthesizing images of an object under novel viewing conditions. Our method requires as few as three images of the object taken under variable illumination, but from a fixed viewpoint. Unlike multi-view based image synthesis, our method does not require the determination of point or line correspondences. Furthermore, our method is able to synthesize not simply novel viewpoints, but novel illumination conditions as well. We demonstrate the effectiveness of our approach by generating synthetic images of human faces.

[1]  David J. Kriegman,et al.  What is the set of images of an object under all possible lighting conditions? , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  David W. Jacobs,et al.  Linear fitting with missing data: applications to structure-from-motion and to characterizing intensity images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Robert J. Woodham,et al.  Analysing Images of Curved Surfaces , 1981, Artif. Intell..

[4]  Jitendra Malik,et al.  Recovering photometric properties of architectural scenes from photographs , 1998, SIGGRAPH.

[5]  Michael J. Brooks,et al.  The variational approach to shape from shading , 1986, Comput. Vis. Graph. Image Process..

[6]  Jan J. Koenderink,et al.  Affine structure and photometry , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Amnon Shashua,et al.  Novel view synthesis in tensor space , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Berthold K. P. Horn,et al.  Determining Shape and Reflectance Using Multiple Images , 1978 .

[9]  Olivier D. Faugeras,et al.  3-D scene representation as a collection of images , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[10]  Amnon Shashua,et al.  Image-based view synthesis by combining trilinear tensors and learning techniques , 1997, VRST '97.

[11]  J J Koenderink,et al.  Affine structure from motion. , 1991, Journal of the Optical Society of America. A, Optics and image science.

[12]  Kiriakos N. Kutulakos,et al.  Shape and Motion of 3-D Curves from Multi – View Image Sequences , 1998 .

[13]  Azriel Rosenfeld,et al.  Computer Vision , 1988, Adv. Comput..

[14]  A. Shashua.  Geometry and Photometry in 3D Visual Recognition , 1992 .

[15]  David J. Kriegman,et al.  Illumination cones for recognition under variable lighting: faces , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[16]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[17]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[18]  Eamon B. Barrett,et al.  Some invariant linear methods in photogrammetry and model-matching , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[20]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[21]  George T. Chou.  Multi-Image Correspondence using Geometric and Structural Constraints , 1997 .

[22]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[24]  Harry Shum,et al.  Principal Component Analysis with Missing Data and Its Application to Polyhedral Object Modeling , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Hideki Hayakawa.  Photometric stereo under a light source with arbitrary motion , 1994 .

[26]  Leonard McMillan,et al.  Post-rendering 3D warping , 1997, SI3D.

[27]  Daniel Snow,et al.  Shape and albedo from multiple images using integrability , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.