Multi-view synthesis: A novel view creation approach for free viewpoint video

Abstract Interactive audio-visual applications such as free viewpoint video (FVV) endeavour to provide unrestricted spatio-temporal navigation within a multiple camera environment. Current novel view creation approaches for scene navigation within FVV applications are either purely image-based, implying large information redundancy and dense sampling of the scene; or involve reconstructing complex 3-D models of the scene. In this paper we present a new multiple image view synthesis algorithm for novel view creation that requires only implicit scene geometry information. The multi-view synthesis approach can be used in any multiple camera environment and is scalable, as virtual views can be created given 1 to N of the available video inputs, providing a means to gracefully handle scenarios where camera inputs decrease or increase over time. The algorithm identifies and selects only the best quality surface areas from available reference images, thereby reducing perceptual errors in virtual view reconstruction. Experimental results are provided and verified using both objective (PSNR) and subjective comparisons and also the improvements over the traditional multiple image view synthesis approach of view-oriented weighting are presented.

[1]  Michael S. Landy,et al.  Computational models of visual processing , 1991 .

[2]  Iso Iec Jtc Sc Wg.  Introduction to Multi-view Video Coding , 2005 .

[3]  Olivier D. Faugeras,et al.  What can two images tell us about a third one? , 1994, International Journal of Computer Vision.

[4]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[5]  Leonard McMillan,et al.  Computing Visibility Without Depth , 1995 .

[6]  Aljoscha Smolic,et al.  3DAV exploration of video-based rendering technology in MPEG , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[7]  Harry Shum,et al.  Review of image-based rendering techniques , 2000, Visual Communications and Image Processing.

[8]  Jed Lengyel,et al.  The Convergence of Graphics and Vision , 1998, Computer.

[9]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[10]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[11]  Paul S. Heckbert,et al.  Fundamentals of Texture Mapping and Image Warping , 1989 .

[12]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[13]  Oliver Schreer,et al.  IMAGE ANALYSIS AND SYNTHESIS FOR MULTIPLE STEREO CAMERA TELECONFERENCING SYSTEMS , 2003 .

[14]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[15]  Jr. Leonard McMillan,et al.  An Image-Based Approach to Three-Dimensional Computer Graphics , 1997 .

[16]  Oliver Schreer,et al.  Image-Based Rendering for Teleconference Systems , 2002, WSCG.

[17]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[18]  Amnon Shashua,et al.  Algebraic Functions For Recognition , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Oliver Schreer,et al.  A modular approach to virtual view creation for a scalable immersive teleconferencing configuration , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[20]  Peter Eisert,et al.  Free viewpoint video extraction, representation, coding, and rendering , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[21]  Oliver Schreer,et al.  3D analysis and image-based rendering for immersive TV applications , 2002, Signal Process. Image Commun..

[22]  Aljoscha Smolic,et al.  Multi-texture modeling of 3D traffic scenes , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[23]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[24]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[25]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[26]  Noel E. O'Connor,et al.  Multiple image view synthesis for free viewpoint video applications , 2005, IEEE International Conference on Image Processing 2005.

[27]  Hans-Peter Seidel,et al.  Model-Based Free-Viewpoint Video Acquisition, Rendering and Encoding , 2004 .

[28]  George Wolberg,et al.  Digital image warping , 1990 .

[29]  Masayuki Tanimoto,et al.  FTV (free viewpoint television) creating ray-based image engineering , 2005, IEEE International Conference on Image Processing 2005.