Color Correction for Immersive Video Applications

In this paper, the color correction method developed for immersive video systems is presented. The proposed method significantly increases the consistency of color characteristics of multiview sequences, understood both as the temporal and the inter-view consistency, what highly improves the subjective quality of the synthesized virtual views presented to the final user of the immersive video system. Moreover, the proposal allows to significantly increase the quality of the depth maps calculated for natural sequences, e.g., for views with colors inconsistent due to different lighting conditions. It enables more efficient compression of natural multiview video, as the newest encoding standards highly depend on the quality of depth maps. In order to evaluate the performance of the proposal, three experiments were conducted. In the first one, the proposal was compared to state of the art in the typical immersive video application – color correction of a natural multiview sequence. In the second experiment, the performance of the proposal was tested on the Middlebury stereo dataset. In both experiments, the quality of synthesized virtual views was assessed subjectively by a group of 70 naïve viewers. The third experiment assessed the influence of color correction on the quality of estimated depth maps. All the experiments showed that the proposal significantly increases the color consistency of the multiview content. Due to the high usefulness and robustness, the proposed color correction method became the MPEG reference software for color correction. The implementation of the method is available for other researchers on the public repository.

[1]  Li Yao,et al.  Fast and high-quality virtual view synthesis from multi-view plus depth videos , 2019, Multimedia Tools and Applications.

[2]  Kamel Mohamed Faraoun,et al.  Feature-Based Color Correction of Multiview Video for Coding and Rendering Enhancement , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Mark E. Payton,et al.  Contrasting Diversity Values: Statistical Inferences Based on Overlapping Confidence Intervals , 2013, PloS one.

[4]  D. Foster Color constancy , 2011, Vision Research.

[5]  Takanori Senoh,et al.  New visual coding exploration in MPEG: Super-MultiView and Free Navigation in Free viewpoint TV , 2016, SD&A.

[6]  Olgierd Stankiewicz,et al.  Depth Map Estimation for Free-Viewpoint Television and Virtual Navigation , 2020, IEEE Access.

[7]  Shao-Yi Chien,et al.  Point-based model construction for free-viewpoint TV , 2013, 2013 IEEE Third International Conference on Consumer Electronics ¿ Berlin (ICCE-Berlin).

[8]  Krzysztof Wegner,et al.  A practical approach to acquisition and processing of free viewpoint video , 2015, 2015 Picture Coding Symposium (PCS).

[9]  Jie Li,et al.  Fast and Adaptive 3D Reconstruction With Extensively High Completeness , 2017, IEEE Transactions on Multimedia.

[10]  Oliver Schreer,et al.  Three-dimensional image processing in the future of immersive media , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Adam Grzelka,et al.  Segmentation-based Method of Increasing The Depth Maps Temporal Consistency , 2018 .

[12]  Adrian Munteanu,et al.  Color correction for large-baseline multiview video , 2017, Signal Process. Image Commun..

[13]  HoraudRadu,et al.  An overview of depth cameras and range scanners based on time-of-flight technologies , 2016 .

[14]  Adrian Hilton,et al.  Proj ective Surface Refinement for Free-Viewpoint Video , 2006 .

[15]  Ran Li,et al.  Light Field Reconstruction With a Hybrid Sparse Regularization-Pseudo 4DCNN Framework , 2020, IEEE Access.

[16]  Yuzhen Niu,et al.  Visually Consistent Color Correction for Stereoscopic Images and Videos , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  W.E. Snyder,et al.  Color image processing pipeline , 2005, IEEE Signal Processing Magazine.

[18]  Marcelo Bertalmío,et al.  Color Matching Images With Unknown Non-Linear Encodings , 2020, IEEE Transactions on Image Processing.

[19]  Toshiaki Fujii,et al.  FTV for 3-D Spatial Communication , 2012, Proceedings of the IEEE.

[20]  Hsu-Feng Hsiao,et al.  A depth refinement algorithm for multi-view video synthesis , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[21]  Rafal Mantiuk,et al.  Comparison of Four Subjective Methods for Image Quality Assessment , 2012, Comput. Graph. Forum.

[22]  Aljoscha Smolic,et al.  3D video objects for interactive applications , 2005, 2005 13th European Signal Processing Conference.

[23]  Toshiaki Fujii,et al.  Multiview Video Coding Using View Interpolation and Color Correction , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[25]  Rufael Mekuria,et al.  Point-Cloud Compression: Moving Picture Experts Group's New Standard in 2020 , 2019, IEEE Consumer Electronics Magazine.

[26]  Olgierd Stankiewicz,et al.  Virtual View Synthesis for 3DoF+ Video , 2019, 2019 Picture Coding Symposium (PCS).

[27]  Adrian Dziembowski,et al.  Real-Time CPU-Based Virtual View Synthesis , 2018, 2018 International Conference on Signals and Electronic Systems (ICSES).

[28]  Olgierd Stankiewicz,et al.  A Free-Viewpoint Television System for Horizontal Virtual Navigation , 2018, IEEE Transactions on Multimedia.

[29]  Gauthier Lafruit,et al.  Robust Multiview Synthesis for Wide-Baseline Camera Arrays , 2018, IEEE Transactions on Multimedia.

[30]  Marek Domanski,et al.  ADAPTIVE COLOR CORRECTION IN VIRTUAL VIEW SYNTHESIS , 2018, 2018 - 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[31]  Graham Finlayson,et al.  Rank-based camera spectral sensitivity estimation. , 2016, Journal of the Optical Society of America. A, Optics, image science, and vision.

[32]  Jean-Bernard Martens,et al.  Subjective quality assessment of compressed images , 1997, Signal Process..

[33]  Yo-Sung Ho,et al.  Color correction using 3D multi-view geometry , 2015, Electronic Imaging.

[34]  André Kaup,et al.  Histogram-Based Prefiltering for Luminance and Chrominance Compensation of Multiview Video , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[36]  Adrian Dziembowski,et al.  Depth Map Refinement for Immersive Video , 2021, IEEE Access.

[37]  Krzysztof Wegner,et al.  Nonlinear depth representation for 3D video coding , 2013, 2013 IEEE International Conference on Image Processing.

[38]  Panos Nasiopoulos,et al.  Virtual View Color Estimation for Free Viewpoint TV Applications Using Gaussian Mixture Model , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[39]  Wei Zhong,et al.  Deep Auxiliary Learning for Point Cloud Generation , 2020, IEEE Access.

[40]  Marek Domanski,et al.  Depth map inter-view consistency refinement for multiview video , 2012, 2012 Picture Coding Symposium.

[41]  Peter Schelkens,et al.  Globally optimized multiview video color correction using dense spatio-temporal matching , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[42]  Chris Johns,et al.  Virtual Reality Industry Forum's View on State of the Immersive Media Industry , 2019 .

[43]  Krzysztof Wegner,et al.  Demonstration of a simple free viewpoint television system , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[44]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[45]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[46]  Mark S. Drew,et al.  Separating a Color Signal into Illumination and Surface Reflectance Components: Theory and Applications , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  Peter Schelkens,et al.  Spatio-Temporally Consistent Color and Structure Optimization for Multiview Video Color Correction , 2015, IEEE Transactions on Multimedia.

[48]  Thomas Wiegand,et al.  3-D Video Representation Using Depth Maps , 2011, Proceedings of the IEEE.

[49]  Krzysztof Wegner,et al.  Immersive visual media — MPEG-I: 360 video, virtual navigation and beyond , 2017, 2017 International Conference on Systems, Signals and Image Processing (IWSSIP).

[50]  Zhan Ma,et al.  Multi-Camera Color Correction via Hybrid Histogram Matching , 2020 .

[51]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[52]  Brian A. Wandell,et al.  The Synthesis and Analysis of Color Images , 1992, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Erhan Ekmekcioglu,et al.  Content Adaptive Enhancement of Multi-View Depth Maps for Free Viewpoint Video , 2011, IEEE Journal of Selected Topics in Signal Processing.

[54]  Shuyuan Yang,et al.  Color Correction and Depth-Based Hierarchical Hole Filling in Free Viewpoint Generation , 2019, IEEE Transactions on Broadcasting.

[55]  Petros Daras,et al.  Temporal and color consistent disparity estimation in stereo videos , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[56]  Wei Xu,et al.  Performance evaluation of color correction approaches for automatic multi-view image and video stitching , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.