Perception-oriented video coding based on image analysis and completion: A review

Perception-oriented video coding based on image analysis and completion (IAC) has received increasing attention over recent years. Several approaches have been proposed that follow diverging strategies: from full compatibility to hybrid block transform coding to alternative codec designs. In this paper, we review the most significant issues in IAC coding and their relevance for future codec designs. The most promising pathways will be analyzed along with limitations and open issues. The challenges in IAC-based video coding will be summarized using methods developed by the authors and others for block-based and region-based IAC coding.

[1]  Jeremy S. De Bonet,et al.  Multiresolution sampling procedure for analysis and synthesis of texture images , 1997, SIGGRAPH.

[2]  Edward J. Delp,et al.  Perceptual quality evaluation for texture and motion based video coding , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[3]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[4]  Jenny Benois-Pineau,et al.  Joint contour-based and motion-based image sequence segmentation for TV image coding at very low bit rate , 1994, Other Conferences.

[5]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[6]  Ingrid Heynderickx,et al.  Studying the added value of visual attention in objective image quality metrics based on eye movement data , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[7]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[8]  Ferran Marqués,et al.  Segmentation-based video coding system allowing the manipulation of objects , 1997, IEEE Trans. Circuits Syst. Video Technol..

[9]  Edward H. Adelson,et al.  Subband texture synthesis for image coding , 1998, Electronic Imaging.

[10]  Homer H. Chen,et al.  Perceptual Rate-Distortion Optimization Using Structural Similarity Index as Quality Metric , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Gary J. Sullivan,et al.  Rate-distortion optimization for video compression , 1998, IEEE Signal Process. Mag..

[12]  Xiaoyan Sun,et al.  Video Coding with Spatio-Temporal Texture Synthesis , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[13]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[14]  Barry G. Haskell,et al.  An encoder-decoder texture replacement method with application to content-based movie coding , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Stefano Soatto,et al.  Dynamic Textures , 2003, International Journal of Computer Vision.

[16]  Gary J. Sullivan,et al.  Special Section on the Joint Call for Proposals on High Efficiency Video Coding (HEVC) Standardization , 2010, IEEE Trans. Circuits Syst. Video Technol..

[17]  Mathias Wien,et al.  Extended Texture Prediction for H.264/AVC Intra Coding , 2007, 2007 IEEE International Conference on Image Processing.

[18]  Santanu Chaudhury,et al.  Parametric Video Compression Scheme Using AR Based Texture Synthesis , 2008, 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing.

[19]  Thiow Keng Tan,et al.  Intra Prediction by Averaged Template Matching Predictors , 2007, 2007 4th IEEE Consumer Communications and Networking Conference.

[20]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[21]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[22]  Ferran Marques,et al.  Hierarchical image sequence model for segmentation: application to region-based sequence coding , 1994, Other Conferences.

[23]  Mohammed Ghanbari,et al.  The accuracy of PSNR in predicting video quality for different video scenes and frame rates , 2012, Telecommun. Syst..

[24]  Thomas Wiegand,et al.  Texture Synthesis Method for Generic Video Sequences , 2007, 2007 IEEE International Conference on Image Processing.

[25]  King Ngi Ngan,et al.  Perceptual video coding: Challenges and approaches , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[26]  Hao Chen,et al.  Video coding using dynamic texture synthesis , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[27]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[28]  Matthew G. Reyes,et al.  Structural texture similarity metrics for retrieval applications , 2008, 2008 15th IEEE International Conference on Image Processing.

[29]  Edward J. Delp,et al.  Spatial Texture Models for Video Compression , 2007, 2007 IEEE International Conference on Image Processing.

[30]  Gary J. Sullivan,et al.  Recent developments in standardization of high efficiency video coding (HEVC) , 2010, Optical Engineering + Applications.

[31]  King Ngi Ngan,et al.  Recent advances in rate control for video coding , 2007, Signal Process. Image Commun..

[32]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[33]  Zhou Wang,et al.  Complex Wavelet Structural Similarity: A New Image Similarity Index , 2009, IEEE Transactions on Image Processing.

[34]  Fan Zhang,et al.  Enhanced video compression with region-based texture models , 2010, 28th Picture Coding Symposium.

[35]  Thomas Wiegand,et al.  Towards efficient intra prediction based on image inpainting methods , 2010, 28th Picture Coding Symposium.

[36]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[37]  Guillermo Sapiro,et al.  Filling-in by joint interpolation of vector fields and gray levels , 2001, IEEE Trans. Image Process..

[38]  Yoshinori Suzuki,et al.  Inter frame coding with template matching spatio-temporal prediction , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[39]  Mauro Barni,et al.  dsp Forum - Future of Video Coding and Transmission , 2006, IEEE Signal Processing Magazine.

[40]  Thomas Wiegand,et al.  Automatic Structure-Aware Inpainting for Complex Image Content , 2008, ISVC.

[41]  Dong Liu,et al.  Edge-Oriented Uniform Intra Prediction , 2008, IEEE Transactions on Image Processing.

[42]  Thomas Wiegand,et al.  Improved video segmentation through robust statistics and MPEG-7 features , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[43]  Philip A. Chou,et al.  Optimal pruning with applications to tree-structured source coding and modeling , 1989, IEEE Trans. Inf. Theory.

[44]  Zhou Wang,et al.  Spatial Pooling Strategies for Perceptual Image Quality Assessment , 2006, 2006 International Conference on Image Processing.

[45]  David R. Bull,et al.  Combined morphological-spectral unsupervised image segmentation , 2005, IEEE Transactions on Image Processing.

[46]  C.-C. Jay Kuo,et al.  Synthesis-based texture coding for video compression with side information , 2008, 2008 15th IEEE International Conference on Image Processing.

[47]  David L. Neuhoff,et al.  Image analysis and compression: renewed focus on texture , 2010, Electronic Imaging.

[48]  Mohammad Sadegh Helfroush,et al.  Image compression based on spatial redundancy removal and image inpainting , 2010, Journal of Zhejiang University SCIENCE C.

[49]  T. Wiegand,et al.  Video coding using closed-loop texture analysis and synthesis , 2011 .

[50]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[51]  Paul A. Ardis,et al.  Inpainting quality assessment , 2010, J. Electronic Imaging.

[52]  Jens-Rainer Ohm,et al.  Dynamic texture synthesis for H.264/AVC inter coding , 2008, 2008 15th IEEE International Conference on Image Processing.

[53]  Yi Guo,et al.  Priority-based template matching intra prediction , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[54]  Murat Kunt,et al.  Very low bit-rate video coding using multi-criterion segmentation , 1994, Proceedings of 1st International Conference on Image Processing.

[55]  Thiow Keng Tan,et al.  Synthesis-in-the-loop for video texture coding , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[56]  K. R. Rao,et al.  Digital Video Image Quality and Perceptual Coding (Signal Processing and Communications) , 2005 .

[57]  Jens-Rainer Ohm,et al.  Multimedia Communication Technology , 2004 .

[58]  Murat Kunt,et al.  Image sequence coding by split and merge , 1991, IEEE Trans. Commun..

[59]  Patrick Bouthemy,et al.  Motion segmentation and qualitative dynamic scene analysis from an image sequence , 1993, International Journal of Computer Vision.

[60]  David L. Neuhoff,et al.  Structural similarity metrics for texture analysis and retrieval , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[61]  Chiew Tuan Kiang.  Novel block-based motion estimation and segmentation for video coding , 2004 .

[62]  Rajiv Soundararajan,et al.  Study of Subjective and Objective Quality Assessment of Video , 2010, IEEE Transactions on Image Processing.

[63]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[64]  Zhou Wang,et al.  Modern Image Quality Assessment , 2006, Modern Image Quality Assessment.

[65]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[66]  Fan Zhang,et al.  A Parametric Framework for Video Compression Using Region-Based Texture Models , 2011, IEEE Journal of Selected Topics in Signal Processing.

[67]  David R. Bull,et al.  Unsupervised image compression-by-synthesis within a JPEG framework , 2008, 2008 15th IEEE International Conference on Image Processing.

[68]  Xiaoyan Sun,et al.  Video coding with spatio-temporal texture synthesis and edge-based inpainting , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[69]  Antonio Ortega,et al.  Rate-distortion methods for image and video compression , 1998, IEEE Signal Process. Mag..

[70]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[71]  N. Kingsbury.  Complex Wavelets for Shift Invariant Analysis and Filtering of Signals , 2001 .