Transmission Policy Selection for Multi-View Content Delivery Over Bandwidth Constrained Channels

I formulate an optimization framework for computing the transmission actions of streaming multi-view video content over bandwidth constrained channels. The optimization finds the schedule for sending the packetized data that maximizes the reconstruction quality of the content, for the given network bandwidth. Two prospective multi-view content representation formats are considered: 1) MVC and 2) video plus depth. In the case of each, I formulate directed graph models that characterize the interdependencies between the data units that comprise the content. For the video plus depth format, I develop a novel space-time error concealment strategy that reconstructs the missing content based on received data units from multiple views. I design multiple techniques to solve the optimization problem of interest, at varying degrees of complexity and accuracy. In conjunction, I derive spatiotemporal models of the reconstruction error for the multi-view content that I employ to reduce the computational requirements of the optimization. I study the performance of my framework via simulation experiments. Significant gains in terms of rate-distortion efficiency are demonstrated over various reference methods.

[1]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[2]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[3]  A. Murat Tekalp,et al.  End-to-end stereoscopic video streaming with content-adaptive rate and format control , 2007, Signal Process. Image Commun..

[4]  Gene Cheung,et al.  Bit allocation for multiview image compression using cubic synthesized view distortion model , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[5]  Ian F. Akyildiz,et al.  A Spatial Correlation Model for Visual Information in Wireless Multimedia Sensor Networks , 2009, IEEE Transactions on Multimedia.

[6]  Thomas Schierl,et al.  Transport and Storage Systems for 3-D Video Using MPEG-2 Systems, RTP, and ISO File Format , 2011, Proceedings of the IEEE.

[7]  A. Murat Tekalp,et al.  Client-Driven Selective Streaming of Multiview Video for Interactive 3DTV , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[8]  I. Feldmann,et al.  3 DPRESENCE – A SYSTEM CONCEPT FOR MULTI-USER AND MULTI-PARTY IMMERSIVE 3 D VIDEOCONFERENCING , 2008 .

[9]  Mihaela van der Schaar,et al.  Information-Constrained Resource Allocation in Multicamera Wireless Surveillance Networks , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Gan-Cheih Chang,et al.  Multi-view image compression and intermediate view synthesis for stereoscopic applications , 2000, 2000 IEEE International Symposium on Circuits and Systems. Emerging Technologies for the 21st Century. Proceedings (IEEE Cat No.00CH36353).

[11]  Bernd Girod,et al.  Analysis of packet loss for compressed video: does burst-length matter? , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[12]  Amarnath Mukherjee,et al.  On the Dynamics and Significance of Low Frequency Components of Internet Load , 1992 .

[13]  S. Burak Gokturk,et al.  A Time-Of-Flight Depth Sensor - System Description, Issues and Solutions , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[14]  Antonio Ortega,et al.  Interactive Streaming of Stored Multiview Video Using Redundant Frame Structures , 2011, IEEE Transactions on Image Processing.

[15]  Gary J. Sullivan,et al.  Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard , 2011, Proceedings of the IEEE.

[16]  Bernd Girod,et al.  Rate-adaptive codes for distributed source coding , 2006, Signal Process..

[17]  Bernd Girod,et al.  Distortion chains for predicting the video distortion for general packet loss patterns , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[18]  Thomas Maugey,et al.  Correlation-aware packet scheduling for multi-camera streaming , 2012, 2012 19th International Packet Video Workshop (PV).

[19]  A. Murat Tekalp,et al.  Adaptive Streaming Of Scalable Stereoscopic Video Over DCCP , 2007 .

[20]  Ze-Nian Li,et al.  Review and Preview: Disocclusion by Inpainting for Image-Based Rendering , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[21]  Harvey J. Everett Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources , 1963 .

[22]  Pascal Frossard,et al.  Popularity-aware rate allocation in multiview video , 2010, Visual Communications and Image Processing.

[23]  Aljoscha Smolic,et al.  The effects of multiview depth video compression on multiview rendering , 2009, Signal Process. Image Commun..

[24]  Aljoscha Smolic,et al.  Multi-View Video Plus Depth Representation and Coding , 2007, 2007 IEEE International Conference on Image Processing.

[25]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[26]  Zhong Zhou,et al.  Real-time stereo-vision system for 3D teleimmersive collaboration , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[27]  Philip A. Chou,et al.  Rate-distortion optimized streaming of packetized media , 2006, IEEE Transactions on Multimedia.

[28]  Béatrice Pesquet-Popescu,et al.  Depth-aided image inpainting for novel view synthesis , 2010, 2010 IEEE International Workshop on Multimedia Signal Processing.

[29]  Toshiaki Fujii,et al.  Free-Viewpoint TV , 2011, IEEE Signal Processing Magazine.

[30]  W. Richard Stevens,et al.  TCP/IP Illustrated, Volume 1: The Protocols , 1994 .

[31]  Bernd Girod,et al.  Systematic Lossy Error Protection of Video Signals , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  Peter Eisert,et al.  3DPresence -A System Concept for Multi-User and Multi-Party Immersive 3D Videoconferencing , 2008 .