Fast Video Quality Enhancement using GANs

Video compression algorithms result in a reduction of image quality, because of their lossy approach to reduce the required bandwidth. This affects commercial streaming services such as Netflix, or Amazon Prime Video, but affects also video conferencing and video surveillance systems. In all these cases it is possible to improve the video quality, both for human view and for automatic video analysis, without changing the compression pipeline, through a post-processing that eliminates the visual artifacts created by the compression algorithms. Generative Adversarial Networks have obtained extremely high quality results in image enhancement tasks; however, to obtain such results large generators are usually employed, resulting in high computational costs and processing time. In this work we present an architecture that can be used to reduce the computational cost and that has been implemented on mobile devices. A possible application is to improve video conferencing, or live streaming. In these cases there is no original uncompressed video stream available. Therefore, we report results using no-reference video quality metric showing high naturalness and quality even for efficient networks.

[1]  Michael S. Brown,et al.  A Contrast Enhancement Framework with JPEG Artifacts Suppression , 2014, ECCV.

[2]  Alberto Del Bimbo,et al.  Deep Universal Generative Adversarial Compression Artifact Removal , 2019, IEEE Transactions on Multimedia.

[3]  Weisi Lin,et al.  Maximum a Posterior and Perceptually Motivated Reconstruction Algorithm: A Generic Framework , 2017, IEEE Transactions on Multimedia.

[4]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[5]  Pavel Zemcík,et al.  Compression Artifacts Removal Using Convolutional Neural Networks , 2016, J. WSCG.

[6]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Wen Gao,et al.  CONCOLOR: Constrained Non-Convex Low-Rank Model for Image Deblocking , 2016, IEEE Transactions on Image Processing.

[9]  Alan C. Bovik,et al.  A Completely Blind Video Integrity Oracle , 2016, IEEE Transactions on Image Processing.

[10]  Chia-Hung Yeh,et al.  Learning-Based Joint Super-Resolution and Deblocking for a Highly Compressed Image , 2015, IEEE Transactions on Multimedia.

[11]  Luca Benini,et al.  CAS-CNN: A deep convolutional neural network for image compression artifact suppression , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[12]  Alberto Del Bimbo,et al.  Deep Generative Adversarial Compression Artifact Removal , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[13]  Xiaohai He,et al.  An Iterative Framework of Cascaded Deblocking and Superresolution for Compressed Images , 2018, IEEE Transactions on Multimedia.

[14]  Charles A. Bouman,et al.  A Document Image Model and Estimation Algorithm for Optimized JPEG Decompression , 2009, IEEE Transactions on Image Processing.

[15]  Nojun Kwak,et al.  Image Restoration by Estimating Frequency Distribution of Local Patches , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Karen O. Egiazarian,et al.  Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images , 2007, IEEE Transactions on Image Processing.

[17]  Yu-Bin Yang,et al.  Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections , 2016, NIPS.

[18]  Truong Q. Nguyen,et al.  Blocking artifact free inverse discrete cosine transform , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[19]  Qing Ling,et al.  D3: Deep Dual-Domain Based Fast Restoration of JPEG-Compressed Images , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Wen Gao,et al.  Compression Artifact Reduction by Overlapped-Block Transform Coefficient Estimation With Block Similarity , 2013, IEEE Transactions on Image Processing.

[22]  Michael Elad,et al.  Postprocessing of Compressed Images via Sequential Denoising , 2015, IEEE Transactions on Image Processing.

[23]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).