New Coding Tools for Illumination and Focus Mismatch Compensation in Multiview Video Coding

We propose new tools for multiview video coding (MVC) that aim to compensate for mismatches between video frames corresponding to different views. Such mismatches could be caused by different shooting positions of the cameras and/or heterogeneous camera settings. In particular, we consider illumination and focus mismatches across views, i.e., such that different portions of a video frame can undergo different illumination and blurriness/sharpness changes with respect to the corresponding areas in frames from the other views. Models for illumination and focus mismatches are proposed and new coding tools are developed from the models. We propose a block-based illumination compensation (IC) technique and a depth-dependent adaptive reference filtering (ARF) approach for cross-view prediction in multiview video coding. In IC, disparity field and illumination changes are jointly computed as part of the disparity estimation search. IC can be adaptively applied by taking into account the rate-distortion characteristics of each block. For ARF, the disparity fields are used to estimate scene depth, such that video frames are first divided into regions with different scene-depth levels. A 2-D filter is then selected for each scene-depth level. These filters are chosen to minimize residual energy, with the goal of compensating for focus mismatches. The resulting filters are applied to the reference frames to generate better matches for cross-view prediction. Furthermore, we propose a coding system that combines IC and ARF. Adjustments are made so as to maximize the gains achieved by using both coding tools, while reducing the complexity of the final integrated system. We analyze the complexity of all proposed methods and present simulation results of IC, ARF and combined system for different multiview sequences based on the H.264/AVC reference codec. When applying the proposed tool to cross-view coding we observe gains of up 1.3 dB as compared to directly using an H.264/AVC codec to perform predictive coding across views.

[1]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[2]  Antonio Ortega,et al.  Adaptive filtering for cross-view prediction in multi-view video coding , 2007, Electronic Imaging.

[3]  Jill M. Boyce,et al.  Weighted prediction in the H.264/MPEG AVC video coding standard , 2004, 2004 IEEE International Symposium on Circuits and Systems (IEEE Cat. No.04CH37512).

[4]  Antonio Ortega,et al.  Block-based Illumination Compensation and Search Techniques for Multiview Video Coding , 2004 .

[5]  Hsien-Che Lee.  Review of image-blur models in a photographic system using the principles of optics , 1990 .

[6]  Til Aach,et al.  Disparity-based segmentation of stereoscopic foreground/background image sequences , 1994, IEEE Trans. Commun..

[7]  Sang Hyun Kim,et al.  Fast local motion-compensation algorithm for video sequences with brightness variations , 2003, IEEE Trans. Circuits Syst. Video Technol..

[8]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[9]  Paul Wintz,et al.  Digital image processing (2nd ed.) , 1987 .

[10]  Guizhong Liu,et al.  A fast and accurate video object detection and segmentation method in the compressed domain , 2003, International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003.

[11]  Peng Yin,et al.  Localized weighted prediction for video coding , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[12]  R. Venkatesh Babu,et al.  Video object segmentation: a compressed domain approach , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  J. Ostermann,et al.  Coding of coefficients of two-dimensional non-separable adaptive Wiener interpolation filter , 2005, Visual Communications and Image Processing.

[14]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[15]  Bertrand Chupeau,et al.  Depth-based segmentation , 1997, IEEE Trans. Circuits Syst. Video Technol..

[16]  Minas E. Spetsakis,et al.  Motion Segmentation by EM Clustering of Good Features , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[17]  J. Rissanen.  A UNIVERSAL PRIOR FOR INTEGERS AND ESTIMATION BY MINIMUM DESCRIPTION LENGTH , 1983 .

[18]  Thomas Wedi.  Adaptive interpolation filter for motion compensated prediction , 2002, Proceedings. International Conference on Image Processing.

[19]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[20]  Ebroul Izquierdo.  Disparity/segmentation analysis: matching with an adaptive window and depth-driven segmentation , 1999, IEEE Trans. Circuits Syst. Video Technol..

[21]  Jörn Ostermann,et al.  Motion-and aliasing-compensated prediction using a two-dimensional non-separable adaptive Wiener interpolation filter , 2005, IEEE International Conference on Image Processing 2005.

[22]  Jörn Ostermann,et al.  Locally Adaptive Non-Separable Interpolation Filter for H.264/AVC , 2006, 2006 International Conference on Image Processing.

[23]  Thomas Wedi.  Adaptive interpolation filters and high-resolution displacements for video coding , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[25]  Michael Gilge,et al.  Motion estimation by scene adaptive block matching and illumination correction , 1990, Other Conferences.

[26]  Wen Gao,et al.  Linear transform based motion compensated prediction for luminance intensity changes , 2005, 2005 IEEE International Symposium on Circuits and Systems.

[27]  Heiko Schwarz,et al.  Context-based adaptive binary arithmetic coding in the H.264/AVC video compression standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[28]  Wolfgang Niehsen,et al.  Block motion estimation using orthogonal projection , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[29]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[30]  Madhukar Budagavi.  Video compression using blur compensation , 2005, IEEE International Conference on Image Processing 2005.

[31]  Hiroshi Watanabe,et al.  Global brightness-variation compensation for video coding , 1998, IEEE Trans. Circuits Syst. Video Technol..

[32]  Gary J. Sullivan,et al.  Rate-constrained coder control and comparison of video coding standards , 2003, IEEE Trans. Circuits Syst. Video Technol..

[33]  Yücel Altunbasak,et al.  A fast parametric motion estimation algorithm with illumination and lens distortion correction , 2003, IEEE Trans. Image Process..