Spatio-Temporal View Interpolation

We propose a fully automatic algorithm for view interpolation of a completely non-rigid dynamic event across both space and time. The algorithm operates by combining images captured across space to compute voxel models of the scene shape at each time instant, and images captured across time to compute the "scene flow" between the voxel models. The scene-flow is the non-rigid 3D motion of every point in the scene. To interpolate in time, the voxel models are "flowed" using an appropriate multiple of the scene flow and a smooth surface fit to the result. The novel image is then computed by ray-casting to the surface at the intermediate time instant, following the scene flow to the neighboring time instants, projecting into the input images at those times, and finally blending the results. We use our algorithm to create re-timed slow-motion fly-by movies of dynamic real-world events.

[1]  Donald P. Greenberg,et al.  Improved Computational Methods for Ray Tracing , 1984, TOGS.

[2]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[3]  Jules Bloomenthal,et al.  Convolution surfaces , 1991, SIGGRAPH.

[4]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[5]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[6]  Marc Levoy,et al.  Feature-based volume metamorphosis , 1995, SIGGRAPH.

[7]  Steven M. Seitz,et al.  View morphing , 1996, SIGGRAPH.

[8]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[9]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[10]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[11]  Katsushi Ikeuchi,et al.  Object shape and reflectance modeling from observation , 1997, SIGGRAPH.

[12]  Takeo Kanade,et al.  The 3D Room: Digitizing Time-Varying 3D Events by Synchronized Multiple Video Streams , 1998 .

[13]  Takeo Kanade,et al.  Constructing virtual worlds using dense stereo , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[14]  Charles R. Dyer,et al.  Interpolating view and scene motion by dynamic view morphing , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[15]  James F. O'Brien,et al.  Shape transformation using variational implicit functions , 1999, SIGGRAPH Courses.

[16]  Takeo Kanade,et al.  Shape and motion carving in 6D , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  Amnon Shashua,et al.  On the synthesis of dynamic scenes from reference views , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[18]  Marc Alexa,et al.  As-rigid-as-possible shape interpolation , 2000, SIGGRAPH.

[19]  David E. Breen,et al.  A Level-Set Approach for the Metamorphosis of Solid Models , 2001, IEEE Trans. Vis. Comput. Graph..

[20]  Yaron Caspi,et al.  Increasing Space-Time Resolution in Video , 2002, ECCV.

[21]  Daniel Cohen-Or,et al.  Tricubic Interpolation of Discrete Surfaces for Binary Volumes , 2003, IEEE Trans. Vis. Comput. Graph..

[22]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.