SecondSkin: An interactive method for appearance transfer

SecondSkin estimates an appearance model for an object visible in a video sequence, without the need for complex interaction or any calibration apparatus. This model can then be transferred to other objects, allowing a non‐expert user to insert a synthetic object into a real video sequence so that its appearance matches that of an existing object, and changes appropriately throughout the sequence. As the method does not require any prior knowledge about the scene, the lighting conditions, or the camera, it is applicable to video which was not captured with this purpose in mind. However, this lack of prior knowledge precludes the recovery of separate lighting and surface reflectance information. The SecondSkin appearance model therefore combines these factors. The appearance model does require a dominant light‐source direction, which we estimate via a novel process involving a small amount of user interaction. The resulting model estimate provides exactly the information required to transfer the appearance of the original object to new geometry composited into the same video sequence.

[1]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[2]  James F. Blinn,et al.  Models of light reflection for computer synthesized pictures , 1998 .

[3]  Olivier D. Faugeras,et al.  3-D Reconstruction of Urban Scenes from Image Sequences , 1998, Comput. Vis. Image Underst..

[4]  Katsushi Ikeuchi,et al.  Illumination Distribution from Brightness in Shadows: Adaptive Estimation of Illumination Distribution with Unknown Reflectance Properties in Shadow Regions , 1999, ICCV.

[5]  Christophe Schlick,et al.  A Survey of Shading and Reflectance Models , 1994, Comput. Graph. Forum.

[6]  Robert L. Cook,et al.  A Reflectance Model for Computer Graphics , 1987, TOGS.

[7]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Roberto Cipolla,et al.  A Practical Method for Estimation of Point Light-Sources , 2001, BMVC.

[9]  Gregory J. Ward,et al.  Measuring and modeling anisotropic reflection , 1992, SIGGRAPH.

[10]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[11]  Xiaochun Cao,et al.  Camera calibration and light source orientation from solar shadows , 2007, Comput. Vis. Image Underst..

[12]  Michael Gervautz,et al.  An Image based Measurement System for Anisotropic Reflection , 1996, Comput. Graph. Forum.

[13]  William H. Press,et al.  Numerical recipes in C (2nd ed.): the art of scientific computing , 1992 .

[14]  Tomoyuki Nishita,et al.  A montage method: the overlaying of the computer generated images onto a background photograph , 1986, SIGGRAPH.

[15]  Pushmeet Kohli,et al.  PoseCut: Simultaneous Segmentation and 3D Pose Estimation of Humans Using Dynamic Graph-Cuts , 2006, ECCV.

[16]  Paul Debevec,et al.  Inverse global illumination: Recovering re?ectance models of real scenes from photographs , 1998 .

[17]  Bui Tuong Phong.  Illumination for computer generated pictures , 1975, Commun. ACM.

[18]  André Gagalowicz,et al.  Image-based rendering of diffuse, specular and glossy surfaces from a single image , 2001, SIGGRAPH.

[19]  James F. Blinn,et al.  Models of light reflection for computer synthesized pictures , 1977, SIGGRAPH.

[20]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[21]  Enhua Wu,et al.  Recovery of material under complex illumination conditions , 2004, GRAPHITE '04.

[22]  Philip H. S. Torr,et al.  VideoTrace: rapid interactive scene modelling from video , 2007, ACM Trans. Graph..

[23]  Paul E. Debevec,et al.  Unlighting the Parthenon , 2004, SIGGRAPH '04.

[24]  R. Love.  Surface reflection model estimation from naturally illuminated image sequences , 1997 .

[25]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[26]  J. F. Murray-Coleman,et al.  The Automated Measurement of BRDFs and their Application to Luminaire Modeling , 1990 .

[27]  Roland Hess,et al.  The Essential Blender: Guide to 3D Creation with the Open Source Suite Blender , 2007 .

[28]  George Drettakis,et al.  Interactive Common Illumination for Computer Augmented Reality , 1997, Rendering Techniques.

[29]  George Drettakis,et al.  Interactive Virtual Relighting of Real Scenes , 2000, IEEE Trans. Vis. Comput. Graph..