Improved Light Field Compression Efficiency through BM3D-Based Denoising Using Inter-View Correlation

Multi-view or light field images have recently gained much attraction from academic and commercial fields to create breakthroughs that go beyond simple video-watching experiences. Immersive virtual reality is an important example. High image quality is essential in systems with a near-eye display device. The compression efficiency is also critical because a large amount of multi-view data needs to be stored and transferred. However, noise can be easily generated during image capturing, and these noisy images severely deteriorate both the quality of experience and the compression efficiency. Therefore, denoising is a prerequisite to produce multi-view-based image contents. In this paper, the structural characteristics of linear multi-view images are fully utilized to increase the denoising speed and performance as well as to improve the compression efficiency. Assuming the sequential processes of denoising and compression, multi-view geometry-based denoising is performed keeping the temporal correlation among views. Experimental results show the proposed scheme significantly improves the compression efficiency of denoised views up to 76.05%, maintaining good denoising quality compared to the popular conventional denoise algorithms.

[1]  Wen Gao,et al.  Fast disparity and motion estimation based on correlations for multiview video coding , 2008, IEEE Transactions on Consumer Electronics.

[2]  Stamatios Lefkimmiatis,et al.  Universal Denoising Networks : A Novel CNN Architecture for Image Denoising , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Ze-Nian Li,et al.  Continuous depth map reconstruction from light fields , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[4]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[5]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[6]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[7]  Karen O. Egiazarian,et al.  Video denoising using separable 4D nonlocal spatiotemporal transforms , 2011, Electronic Imaging.

[8]  Namalka Liyanage,et al.  Real-Time Light Field Denoising Using a Novel Linear 4-D Hyperfan Filter , 2020, IEEE Transactions on Circuits and Systems I: Regular Papers.

[9]  Aljoscha Smolic,et al.  Light field denoising by sparse 5D transform domain collaborative filtering , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[10]  Nick Barnes,et al.  Real Image Denoising With Feature Attention , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Marc Lebrun,et al.  An Analysis and Implementation of the BM3D Image Denoising Method , 2012, Image Process. Line.

[12]  Yulan Guo,et al.  DeOccNet: Learning to See Through Foreground Occlusions in Light Fields , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[13]  Jean-Michel Morel,et al.  Model-Blind Video Denoising via Frame-To-Frame Training , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  André Kaup,et al.  Advanced Lagrange Multiplier Selection for Hybrid Video Coding , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[15]  G. Bjontegaard,et al.  Calculation of Average PSNR Differences between RD-curves , 2001 .

[16]  Karen O. Egiazarian,et al.  Video denoising by sparse 3D transform-domain collaborative filtering , 2007, 2007 15th European Signal Processing Conference.

[17]  Chao Li,et al.  Computer Vision and Image Understanding , 2022 .

[18]  Wei An,et al.  Selective Light Field Refocusing for Camera Arrays Using Bokeh Rendering and Superresolution , 2019, IEEE Signal Processing Letters.

[19]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[20]  Gary J. Sullivan,et al.  Rate-distortion optimization for video compression , 1998, IEEE Signal Process. Mag..

[21]  Takayuki Hamamoto,et al.  Fast multiple-view denoising based on image reconstruction by plane sweeping , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[22]  Jungang Yang,et al.  Spatial-Angular Interaction for Light Field Image Super-Resolution , 2020, ECCV.

[23]  Dong Liu,et al.  Light field super-resolution using internal and external similarities , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[24]  Yanjun Li,et al.  Joint Adaptive Sparsity and Low-Rankness on the Fly: An Online Tensor Reconstruction Scheme for Video Denoising , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Jie Chen,et al.  Light Field Denoising via Anisotropic Parallax Analysis in a CNN Framework , 2018, IEEE Signal Processing Letters.

[26]  Jose Luis Lisani,et al.  Patch-Based Video Denoising With Optical Flow Estimation , 2016, IEEE Transactions on Image Processing.

[27]  Yu Hen Hu,et al.  Patch-based multiple view image denoising with occlusion handling , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Hyuk-Jae Lee,et al.  Flexibly Connectable Light Field System For Free View Exploration , 2020, IEEE Transactions on Multimedia.

[29]  Luc Van Gool,et al.  Self-Guided Network for Fast Image Denoising , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  Michael Elad,et al.  Multi-Scale Patch-Based Image Restoration , 2016, IEEE Transactions on Image Processing.

[31]  In-So Kweon,et al.  EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Zhaolin Xiao,et al.  Seeing Beyond Foreground Occlusion: A Joint Framework for SAP-Based Scene Depth and Appearance Reconstruction , 2017, IEEE Journal of Selected Topics in Signal Processing.

[33]  F. Calderon,et al.  Depth map estimation in light fields using an stereo-like taxonomy , 2014, 2014 XIX Symposium on Image, Signal Processing and Artificial Vision.

[34]  Hongrui Jiang,et al.  Multi-View Image Denoising Using Convolutional Neural Network , 2019, Sensors.

[35]  Hyuk-Jae Lee,et al.  Ray-space360: An extension of ray-space for omnidirectional free viewpoint , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[36]  Qionghai Dai,et al.  Learning Sheared EPI Structure for Light Field Reconstruction , 2019, IEEE Transactions on Image Processing.

[37]  Alessandro Foi,et al.  Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering , 2007, IEEE Transactions on Image Processing.

[38]  J. Li,et al.  An Epipolar Geometry-Based Fast Disparity Estimation Algorithm for Multiview Image and Video Coding , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[39]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[40]  Aljoscha Smolic,et al.  Light Field Super-Resolution via LFBM5D Sparse Coding , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).