Video Super-Resolution with Compensation in Feature Extraction

Conventional Convolutional Neural Network (CNN) based video super-resolution (VSR) methods heavily depend on motion compensation. Pixels in input frames are warped according to flow-like information to eliminate inter-frame differences. These methods have to make a trade-off between the distraction caused by spatio-temporal inconsistency and the pixel-wise detail damage caused by compensation. In this paper, we propose a novel video super-resolution method with a dynamic filter network based compensation module and a residual network based SR module. Unlike traditional VSR techniques, our method does not warp input pixels, but performs motion compensation during feature extractions. The experimental results demonstrate that our method outperforms the state-of-the-art VSR algorithms by at least 1.08 dB in terms of PSNR, and recovers more details together with superior visual quality.

[1]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[2]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[3]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Convolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yu-Bin Yang,et al.  Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections , 2016, NIPS.

[6]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Separable Convolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Deqing Sun,et al.  A Bayesian approach to adaptive video super resolution , 2011, CVPR 2011.

[9]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[10]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Christian Ledig,et al.  Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Jiying Zhao,et al.  Video Super-Resolution via Dynamic Local Filter Network , 2018, 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[14]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Renjie Liao,et al.  Detail-Revealing Deep Video Super-Resolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[16]  Zengfu Wang,et al.  Video Superresolution via Motion Compensation and Deep Residual Learning , 2017, IEEE Transactions on Computational Imaging.

[17]  Luc Van Gool,et al.  Dynamic Filter Networks , 2016, NIPS.

[18]  Thomas Brox,et al.  End-to-End Learning of Video Super-Resolution with Motion Compensation , 2017, GCPR.

[19]  Aggelos K. Katsaggelos,et al.  Video Super-Resolution With Convolutional Neural Networks , 2016, IEEE Transactions on Computational Imaging.

[20]  Renjie Liao,et al.  Video Super-Resolution via Deep Draft-Ensemble Learning , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Yu Liu,et al.  Video super-resolution using motion compensation and residual bidirectional recurrent convolutional network , 2017, 2017 IEEE International Conference on Image Processing (ICIP).