FTV for 3-D Spatial Communication

Free-viewpoint TV (FTV) is cutting the frontier of audiovisual communications. FTV is an innovative media that enables us to view 3-D space by freely changing our viewpoints. It also allows us to listen at any listening point in the 3-D space. Since FTV transmits all audiovisual information of the 3-D space, it can reconstruct an audiovisual replica of the 3-D space anywhere and anytime over distance and time. For video, FTV captures a part of rays in 3-D space by using many cameras, and the other rays that are not captured are obtained by interpolating the captured rays. We constructed real-time FTV systems including the complete chain of operation from image capture to display. We also carried out FTV on a laptop computer and a mobile player. For audio, two kinds of free listening-point systems are demonstrated. MPEG regarded FTV as the most challenging 3-D media and has been conducting its international standardization activities. The first phase of FTV was multiview video coding (MVC) and the second phase of FTV is 3-D video (3DV). MVC enables the efficient coding of multiple camera views and was completed in 2009. MVC has been adopted by Blu-ray 3-D. 3DV is a standard that targets serving a variety of 3-D displays and its call for proposals was issued in March 2011.

[1]  Toshiaki Fujii,et al.  Ray-space acquisition system of all-around convergent views using a rotation mirror , 2007, SPIE Optics East.

[2]  Toshiaki Fujii,et al.  Real-time arbitrary view interpolation and rendering system using ray-space , 2005, SPIE Optics East.

[3]  Toshiaki Fujii,et al.  Multipoint Measuring System for Video and Sound - 100-camera and microphone system , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[4]  Toshiyuki Kimura,et al.  Reproduction of sound radiation directivities of musical instruments by a spherical loudspeaker with multiple transducers , 2010, VRCAI '10.

[5]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[6]  Toshiaki Fujii,et al.  Novel view synthesis with residual error feedback for FTV , 2010, Electronic Imaging.

[7]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[8]  Kazuya Takeda,et al.  ON ACOUSTICS MADRID , 2-7 SEPTEMBER 2007 Development of Selectable Viewpoint and Listening Point System for Musical Performance PACS : 43 . 60 , 2007 .

[9]  K. Matsuoka,et al.  Minimal distortion principle for blind source separation , 2002, Proceedings of the 41st SICE Annual Conference. SICE 2002..

[10]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[11]  Toshiaki Fujii,et al.  Probabilistic reliability based view synthesis for FTV , 2010, 2010 IEEE International Conference on Image Processing.

[12]  Kazuya Takeda,et al.  Encoding large array signals into a 3D sound field representation for selective listening point audio based on blind source separation , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[13]  Toshiaki Fujii,et al.  Multiview Video Coding Using View Interpolation and Color Correction , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[14]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[15]  Toshiaki Fujii,et al.  A NOVEL RECTIFICATION METHOD FOR TWO-DIMENSIONAL CAMERA ARRAY BY PARALLELIZING LOCUS OF FEATURE POINTS(INTERNATIONAL Workshop on Advanced Image Technology 2008) , 2007 .

[16]  Masayuki Tanimoto,et al.  Frameworks for FTV coding , 2009, 2009 Picture Coding Symposium.

[17]  Charles T. Loop,et al.  Computing rectifying homographies for stereo vision , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[18]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Toshiaki Fujii,et al.  Free viewpoint TV system based on ray-space representation , 2002, SPIE ITCom.

[20]  Andrew Blake,et al.  Efficient Dense Stereo with Occlusions for New View-Synthesis by Four-State Dynamic Programming , 2006, International Journal of Computer Vision.

[21]  Toshiaki Fujii,et al.  Artifact reduction using reliability reasoning for image generation of FTV , 2010, J. Vis. Commun. Image Represent..

[22]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[23]  Toshiaki Fujii,et al.  3DAV integrated system featuring arbitrary listening-point and viewpoint generation , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[24]  Masayuki Tanimoto,et al.  FTV: Free-viewpoint Television , 2006, Signal Process. Image Commun..

[25]  Toshiaki Fujii,et al.  Free-Viewpoint TV , 2011, IEEE Signal Processing Magazine.

[26]  Toshiaki Fujii,et al.  Arbitrary Listening-Point Generation Using Sub-Band Representation of Sound Wave Ray-Space , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[27]  Toshiaki Fujii,et al.  Colour Correction for Multiple-camera System by Using Correspondences , 2007 .

[28]  Toshiaki Fujii,et al.  View Generation with 3D Warping Using Depth Information for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[29]  Kazuya Takeda,et al.  Blind source separation of musical signals applied to selectable‐listening‐point audio reconstruction , 2006 .

[30]  Toshiaki Fujii,et al.  Experimental system of free viewpoint television , 2003, IS&T/SPIE Electronic Imaging.

[31]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[33]  Aapo Hyvärinen,et al.  A Fast Fixed-Point Algorithm for Independent Component Analysis of Complex Valued Signals , 2000, Int. J. Neural Syst..

[34]  Wei-Chao Chen,et al.  Light field mapping: efficient representation and hardware rendering of surface light fields , 2002, SIGGRAPH.

[35]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Toshiaki Fujii,et al.  Ray space coding for 3D visual communication , 1996 .

[37]  Milan Sonka,et al.  Image Processing, Analysis and Machine Vision , 1993, Springer US.

[38]  Yi Deng,et al.  Stereo Correspondence with Occlusion Handling in a Symmetric Patch-Based Graph-Cuts Model , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Hiroshi Sawada,et al.  A robust and precise method for solving the permutation problem of frequency-domain blind source separation , 2004, IEEE Transactions on Speech and Audio Processing.

[40]  Toshiaki Fujii,et al.  A real-time ray-space acquisition system , 2004, IS&T/SPIE Electronic Imaging.

[41]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[42]  Masayuki Tanimoto,et al.  Introduction to the Special Section on Multiview Video Coding , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[43]  D. Nistér,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[45]  Masayuki Tanimoto.  Overview of free viewpoint television , 2006, Signal Process. Image Commun..

[46]  Toshiaki Fujii,et al.  Free viewpoint image generation using multi-pass dynamic programming , 2007, Electronic Imaging.

[47]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[48]  Toshiaki Fujii,et al.  The Seelinder: Cylindrical 3D display viewable from 360 degrees , 2010, J. Vis. Commun. Image Represent..

[49]  Heung-Yeung Shum,et al.  Image-Based Rendering and Synthesis , 2007, IEEE Signal Processing Magazine.

[50]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Toshiaki Fujii,et al.  Colour correction for multiple-camera system by using correspondences , 2007 .

[52]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[53]  Jian Sun,et al.  Symmetric stereo matching for occlusion handling , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[54]  Masayuki Tanimoto.  Free-Viewpoint Television , 2010, Image and Geometry Processing for 3-D Cinematography.