Residual Highway Convolutional Neural Networks for in-loop Filtering in HEVC

High efficiency video coding (HEVC) standard achieves half bit-rate reduction while keeping the same quality compared with AVC. However, it still cannot satisfy the demand of higher quality in real applications, especially at low bit rates. To further improve the quality of reconstructed frame while reducing the bitrates, a residual highway convolutional neural network (RHCNN) is proposed in this paper for in-loop filtering in HEVC. The RHCNN is composed of several residual highway units and convolutional layers. In the highway units, there are some paths that could allow unimpeded information across several layers. Moreover, there also exists one identity skip connection (shortcut) from the beginning to the end, which is followed by one small convolutional layer. Without conflicting with deblocking filter (DF) and sample adaptive offset (SAO) filter in HEVC, RHCNN is employed as a high-dimension filter following DF and SAO to enhance the quality of reconstructed frames. To facilitate the real application, we apply the proposed method to I frame, P frame, and B frame, respectively. For obtaining better performance, the entire quantization parameter (QP) range is divided into several QP bands, where a dedicated RHCNN is trained for each QP band. Furthermore, we adopt a progressive training scheme for the RHCNN where the QP band with lower value is used for early training and their weights are used as initial weights for QP band of higher values in a progressive manner. Experimental results demonstrate that the proposed method is able to not only raise the PSNR of reconstructed frame but also prominently reduce the bit-rate compared with HEVC reference software.

[1]  Chen-Yi Lee,et al.  An In/Post-Loop Deblocking Filter With Hybrid Filtering Schedule , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  Ali M. Reza,et al.  Combined edge crispiness and statistical differencing for deblocking JPEG compressed images , 2001, IEEE Trans. Image Process..

[3]  Truong Q. Nguyen,et al.  Enhanced Adaptive Loop Filter for Motion Compensated Frame , 2011, IEEE Transactions on Image Processing.

[4]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[5]  Damon M. Chandler,et al.  A perceptual quantization strategy for HEVC based on a convolutional neural network trained on natural images , 2015, SPIE Optical Engineering + Applications.

[6]  Jani Lainema,et al.  Adaptive deblocking filter , 2003, IEEE Trans. Circuits Syst. Video Technol..

[7]  Munchurl Kim,et al.  CNN-based in-loop filtering for coding efficiency improvement , 2016, 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[8]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Takashi Watanabe,et al.  Adaptive Loop Filtering for Video Coding , 2013, IEEE Journal of Selected Topics in Signal Processing.

[10]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[11]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[12]  Thomas Sikora,et al.  Adaptive Temporal Trajectory Filtering for Video Compression , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[14]  Heiko Schwarz,et al.  Transform Coding Techniques in HEVC , 2013, IEEE Journal of Selected Topics in Signal Processing.

[15]  Minhua Zhou,et al.  HEVC Deblocking Filter , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Christine Guillemot,et al.  Robust Video Coding Based on Multiple Description Scalar Quantization With Side Information , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Chia-Yang Tsai,et al.  Sample Adaptive Offset in the HEVC Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[18]  Dong Liu,et al.  A Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding , 2016, MMM.

[19]  Qionghai Dai,et al.  Stereo Interleaving Video Coding With Content Adaptive Image Subsampling , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[20]  Kemal Ugur,et al.  Intra Coding of the HEVC Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Xiaoou Tang,et al.  Compression Artifacts Reduction by a Deep Convolutional Network , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  King Ngi Ngan,et al.  2-D Order-16 Integer Transforms for HD Video Coding , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[27]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[29]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[30]  Yu-Wen Huang,et al.  Motion Vector Coding in the HEVC Standard , 2013, IEEE Journal of Selected Topics in Signal Processing.

[31]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[32]  King Ngi Ngan,et al.  Two-Layer Directional Transform for High Performance Video Coding , 2012, IEEE Transactions on Circuits and Systems for Video Technology.