Performance relighting and reflectance transformation with time-multiplexed illumination

We present a technique for capturing an actor's live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-multiplexed basis lighting conditions, and to record these conditions with a high-speed video camera so that many conditions are recorded in the span of the desired output frame interval. We investigate several lighting bases for representing the sphere of incident illumination using a set of discrete LED light sources, and we estimate and compensate for subject motion using optical flow and image warping based on a set of tracking frames inserted into the lighting basis. To composite the illuminated performance into a new background, we include a time-multiplexed matte within the basis. We also show that the acquired data enables time-varying surface normals, albedo, and ambient occlusion to be estimated, which can be used to transform the actor's reflectance to produce both subtle and stylistic effects.

[1]  Steve Marschner,et al.  Modeling and Rendering for Realistic Facial Animation , 2000, Rendering Techniques.

[2]  Pat Hanrahan,et al.  All-frequency shadows using non-linear wavelet lighting approximation , 2003, ACM Trans. Graph..

[3]  Pat Hanrahan,et al.  A fast relighting engine for interactive cinematic lighting design , 2000, SIGGRAPH.

[4]  Athinodoros S. Georghiades,et al.  Recovering 3-D Shape and Reflectance From a Small Number of Photographs , 2003, Rendering Techniques.

[5]  Shree K. Nayar,et al.  Eyes for relighting , 2004, SIGGRAPH 2004.

[6]  Gregory D. Hager,et al.  Real-time tracking of image regions with changes in geometry and illumination , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[7]  Pat Hanrahan,et al.  Conveying shape and features with image-based relighting , 2003, IEEE Visualization, 2003. VIS 2003..

[8]  Pieter Peers,et al.  Smooth Reconstruction and Compact Representation of Reflectance Functions for Image-based Relighting , 2004, Rendering Techniques.

[9]  Li Zhang,et al.  Spacetime faces: high resolution capture for modeling and animation , 2004, SIGGRAPH 2004.

[10]  Brian A. Barsky,et al.  Advanced Renderman: Creating CGI for Motion Pictures , 1999 .

[11]  Andrew Gardner,et al.  Animatable Facial Reflectance Fields , 2004 .

[12]  Michael J. Black,et al.  A framework for the robust estimation of optical flow , 1993, 1993 (4th) International Conference on Computer Vision.

[13]  Jr. Hall.  Combinatorial theory (2nd ed.) , 1998 .

[14]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[15]  David Salesin,et al.  Environment matting and compositing , 1999, SIGGRAPH.

[16]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[17]  Raymond Fielding,et al.  The technique of special effects cinematography , 1965 .

[18]  Xiaolin Wu.  An efficient antialiasing technique , 1991, SIGGRAPH.

[19]  Marc Levoy,et al.  Real-time 3D model acquisition , 2002, ACM Trans. Graph..

[20]  Jan Kautz,et al.  Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments , 2002 .

[21]  Shree K. Nayar,et al.  A theory of multiplexed illumination , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[22]  Paul E. Debevec,et al.  Acquiring the reflectance field of a human face , 2000, SIGGRAPH.

[23]  Ned Greene,et al.  Environment Mapping and Other Applications of World Projections , 1986, IEEE Computer Graphics and Applications.

[24]  Zicheng Liu,et al.  Face relighting with radiance environment maps , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[25]  Paul E. Debevec,et al.  A photometric approach to digitizing cultural artifacts , 2001, VAST '01.

[26]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[27]  Paul A. Beardsley,et al.  Design galleries: a general approach to setting parameters for computer graphics and animation , 1997, SIGGRAPH.

[28]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[29]  Thomas Malzbender,et al.  Polynomial texture maps , 2001, SIGGRAPH.

[30]  Ramesh Raskar,et al.  Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging , 2004, SIGGRAPH 2004.

[31]  Adam Finkelstein,et al.  Shadows for cel animation , 2000, SIGGRAPH.

[32]  Julie Dorsey,et al.  Effic ient Re-rendering of Naturally Illuminated Environments , 1994 .

[33]  N. Sloane,et al.  Hadamard transform optics , 1979 .

[34]  Irfan A. Essa,et al.  Image-based motion blur for stop motion animation , 2001, SIGGRAPH.

[35]  Hayden Landis,et al.  Production-Ready Global Illumination , 2004 .

[36]  Paul E. Debevec,et al.  Optimizing Color Matching in a Lighting Reproduction System for Complex Subject and Illuminant Spectra , 2003, Rendering Techniques.

[37]  David J. Kriegman,et al.  Illumination-based image synthesis: creating novel images of human faces under differing pose and lighting , 1999, Proceedings IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes (MVIEW'99).

[38]  Andrew Gardner,et al.  A lighting reproduction approach to live-action compositing , 2002, SIGGRAPH.

[39]  Zoli Vidor.  An Infrared Self-Matting Process , 1960 .

[40]  Tom Duff,et al.  Compositing digital images , 1984, SIGGRAPH.