State-of-the Art Motion Estimation in the Context of 3D TV

Progress in image sensors and computation power has fueled studies to improve acquisition, processing, and analysis of 3D streams along with 3D scenes/objects reconstruction. The role of motion compensation/motion estimation (MCME) in 3D TV from end-to-end user is investigated in this chapter. Motion vectors (MVs) are closely related to the concept of disparities, and they can help improving dynamic scene acquisition, content creation, 2D to 3D conversion, compression coding, decompression/decoding, scene rendering, error concealment, virtual/augmented reality handling, intelligent content retrieval, and displaying. Although there are different 3D shape extraction methods, this chapter focuses mostly on shape-from-motion (SfM) techniques due to their relevance to 3D TV. SfM extraction can restore 3D shape information from a single camera data.

[1]  Pieter Peers,et al.  Dynamic shape capture using multi-view photometric stereo , 2009, ACM Trans. Graph..

[2]  Heiko Schwarz,et al.  Analysis of Hierarchical B Pictures and MCTF , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[3]  Mounir Kaaniche,et al.  Dense disparity estimation in multiview video coding , 2009, 2009 IEEE International Workshop on Multimedia Signal Processing.

[4]  Reuben A. Farrugia,et al.  Exploiting depth information for fast multi-view video coding , 2010, 28th Picture Coding Symposium.

[5]  Jean-Yves Guillemaut,et al.  Objective Quality Assessment in Free-Viewpoint Video Production , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[6]  Alex Pentland,et al.  3 D Structure from 2 D Motion , 2001 .

[7]  A. Bovik,et al.  Image Quality Assessment , 2012 .

[8]  S. Burak Gokturk,et al.  A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[9]  Carl J. Debono,et al.  Error concealment techniques for H.264/MVC encoded sequences , 2010 .

[10]  Iain E. G. Richardson,et al.  H.264 and MPEG-4 Video Compression: Video Coding for Next-Generation Multimedia , 2003 .

[11]  Adrian Hilton,et al.  Objective Quality Assessment in Free-Viewpoint Video Production , 2008, 3DTV-CON 2008.

[12]  金谷 健一.  Group-theoretical methods in image understanding , 1990 .

[13]  Wei-Xing Wang,et al.  Automatic Depth Map Estimation of Monocular Indoor Environments , 2008, 2008 International Conference on MultiMedia and Information Technology.

[14]  B. Girod,et al.  Motion and Disparity Compensated Coding for Video Camera Arrays , 2006 .

[15]  Levent Onural,et al.  Three-Dimensional Television: Capture, Transmission, Display , 2007 .

[16]  Bernhard Schölkopf,et al.  How to Find Interesting Locations in Video: A Spatiotemporal Interest Point Detector Learned from Human Eye Movements , 2007, DAGM-Symposium.

[17]  B. Cyganek.  An Introduction to 3D Computer Vision Techniques and Algorithms , 2009 .

[18]  Anthony Vetro,et al.  Extensions of H.264/AVC for Multiview Video Compression , 2006, 2006 International Conference on Image Processing.

[19]  Marc Pollefeys,et al.  An evolutionary and optimised approach on 3D-TV , 2002 .

[20]  Sukhendu Das,et al.  Simulation Studies For The Performance Analysis Of The Reconstruction Of A Line In 3-D From Two Arbitrary Perspective Views Using Two Plane Intersection Method , 2003, Int. J. Comput. Math..

[21]  Wei-Chih Chen,et al.  A 2D to 3D conversion scheme based on depth cues analysis for MPEG videos , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[22]  Michel Dhome,et al.  Real time tracking of 3D objects: an efficient and robust approach , 2002, Pattern Recognit..

[23]  Nir A. Sochen,et al.  A Geometric Framework and a New Criterion in Optical Flow Modeling , 2008, Journal of Mathematical Imaging and Vision.

[24]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[25]  Yannick Morvan,et al.  Design considerations for view interpolation in a 3D video coding framework , 2006 .

[26]  Carl J. Debono,et al.  Resilient Digital Video Transmission over Wireless Channels using Pixel-Level Artefact Detection Mechanisms , 2010 .

[27]  Vania V. Estrela,et al.  Data-Driven Motion Estimation with Spatial Adaptation , 2012 .

[28]  Jean-Bernard Martens,et al.  Multidimensional modeling of image quality , 2002, Proc. IEEE.

[29]  Levent Onural,et al.  Three-Dimensional Television , 2008 .

[30]  Mei Yu,et al.  Comparison of the depth quantification method in terms of coding and synthesizing capacity in 3DTV system , 2008, 2008 9th International Conference on Signal Processing.

[31]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[32]  Toshiaki Fujii,et al.  A semi-automatic multi-view depth estimation method , 2010, Visual Communications and Image Processing.

[33]  Rabab Kreidieh Ward,et al.  An H.264-based scheme for 2D to 3D video conversion , 2009, 2009 Digest of Technical Papers International Conference on Consumer Electronics.

[34]  Aljoscha Smolic,et al.  3DAV exploration of video-based rendering technology in MPEG , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Enrico Magli,et al.  An error concealment algorithm for streaming video , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[36]  Thomas Sikora,et al.  The MPEG-7 visual standard for content description-an overview , 2001, IEEE Trans. Circuits Syst. Video Technol..

[37]  J. Ferreira,et al.  Stereoscopic image rendering based on depth maps created from blur and edge information , 2005, IS&T/SPIE Electronic Imaging.

[38]  Luca Lucchese,et al.  A Frequency Domain Technique for Range Data Registration , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Alberto Del Bimbo,et al.  Encyclopedia of Multimedia , 2006 .

[41]  Vania V. Estrela,et al.  Error concealment by means of clustered blockwise PCA , 2009, 2009 Picture Coding Symposium.

[42]  Masayuki Tanimoto,et al.  3D-TV System with Depth-Image-Based Rendering , 2012 .

[43]  Leonard McMillan,et al.  A Real-Time Distributed Light Field Camera , 2002, Rendering Techniques.

[44]  Ventseslav Sainov,et al.  3-D Time-Varying Scene Capture Technologies—A Survey , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[45]  Euee S. Jang,et al.  An introduction to the MPEG-4 animation framework eXtension , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[46]  Christian Wöhler,et al.  3D Computer Vision - Efficient Methods and Applications , 2009, X.media.publishing.

[47]  Yiannis Aloimonos,et al.  Polydioptric camera design and 3D motion estimation , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[48]  John Oliensis,et al.  A Critique of Structure-from-Motion Algorithms , 2000, Comput. Vis. Image Underst..

[49]  Carl J. Debono,et al.  Error concealment techniques for multi-view video , 2010, 2010 IFIP Wireless Days.

[50]  Hideo Saito,et al.  Augmented reality for 3D TV using depth camera input , 2010, 2010 16th International Conference on Virtual Systems and Multimedia.

[51]  Liang-Gee Chen,et al.  A novel 2Dd-to-3D conversion system using edge information , 2010, IEEE Transactions on Consumer Electronics.

[52]  S. Shankar Sastry,et al.  An Invitation to 3-D Vision , 2004 .

[53]  Bahram Javidi,et al.  Three-Dimensional Television, Video and Display Technology , 2002 .

[54]  B. Krauskopf,et al.  Proc of SPIE , 2003 .

[55]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[56]  Kjell Brunnström,et al.  VQeg validation and ITU standardization of objective perceptual video quality metrics [Standards in a Nutshell] , 2009, IEEE Signal Processing Magazine.

[57]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[58]  Alex Pentland,et al.  3D structure from 2D motion , 1999, IEEE Signal Process. Mag..

[59]  Cordelia Schmid,et al.  Evaluation of Interest Point Detectors , 2000, International Journal of Computer Vision.

[60]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[61]  Béatrice Pesquet-Popescu,et al.  Joint depth-motion dense estimation for multiview video coding , 2010, J. Vis. Commun. Image Represent..

[62]  Holger G. Krapp,et al.  Extracting Egomotion from Optic Flow: Limits of Accuracy and Neural Matched Filters , 2001 .

[63]  Filippo Speranza,et al.  Depth image based rendering for multiview stereoscopic displays: role of information at object boundaries , 2005, SPIE Optics East.

[64]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[65]  Chang-Su Kim,et al.  Multi-camera imaging, coding and innovative display: techniques and systems , 2010, J. Vis. Commun. Image Represent..

[66]  Stefano Soatto,et al.  3-D Shape Estimation and Image Restoration - Exploiting Defocus and Motion Blur , 2006 .

[67]  A. Bovik.  F Mean Squared Error: Love It or Leave It? , 2009 .

[68]  Yael Moses,et al.  Multi-view Scene Flow Estimation: A View Centered Variational Approach , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[69]  Leonard McMillan,et al.  A new reconstruction filter for undersampled light fields , 2003 .

[70]  Ahmet M. Kondoz,et al.  Comparison of stereo video coding support in MPEG-4 MAC, H.264/AVC and H.264/SVC , 2007 .

[71]  Christian Whler.  3D Computer Vision: Efficient Methods and Applications , 2009 .

[72]  Nikolas P. Galatsanos,et al.  Spatially adaptive regularized pel-recursive motion estimation based on the EM algorithm , 2000, Electronic Imaging.

[73]  Aljoscha Smolic,et al.  Interactive 3-D Video Representation and Coding Technologies , 2005, Proceedings of the IEEE.

[74]  André Kaup,et al.  4D Scalable Multi-View Video Coding Using Disparity Compensated View Filtering and Motion Compensated Temporal Filtering , 2006, 2006 IEEE Workshop on Multimedia Signal Processing.

[75]  W. Gao,et al.  Inter-View Direct Mode for Multiview Video Coding , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[76]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[77]  Marcus A. Magnor,et al.  Multi-view coding for image-based rendering using 3-D scene geometry , 2003, IEEE Trans. Circuits Syst. Video Technol..