Invertible Image Rescaling

High-resolution digital images are usually downscaled to fit various display screens or save the cost of storage and bandwidth, meanwhile the post-upscaling is adpoted to recover the original resolutions or the details in the zoom-in images. However, typical image downscaling is a non-injective mapping due to the loss of high-frequency information, which leads to the ill-posed problem of the inverse upscaling procedure and poses great challenges for recovering details from the downscaled low-resolution images. Simply upscaling with image super-resolution methods results in unsatisfactory recovering performance. In this work, we propose to solve this problem by modeling the downscaling and upscaling processes from a new perspective, i.e. an invertible bijective transformation, which can largely mitigate the ill-posed nature of image upscaling. We develop an Invertible Rescaling Net (IRN) with deliberately designed framework and objectives to produce visually-pleasing low-resolution images and meanwhile capture the distribution of the lost information using a latent variable following a specified distribution in the downscaling process. In this way, upscaling is made tractable by inversely passing a randomly-drawn latent variable with the low-resolution image through the network. Experimental results demonstrate the significant improvement of our model over existing methods in terms of both quantitative and qualitative evaluations of image upscaling reconstruction from downscaled images.

[1]  Ullrich Köthe,et al.  Analyzing Inverse Problems with Invertible Neural Networks , 2018, ICLR.

[2]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Dongsu Han,et al.  How will Deep Learning Change Internet Video Delivery? , 2017, HotNets.

[4]  Shanshan Li,et al.  Multi-Scale Invertible Network for Image Super-Resolution , 2019, MMAsia.

[5]  Nicola Asuni,et al.  Submitted to Ieee Transactions on Image Processing 1 Real Time Artifact-free Image Upscaling , 2022 .

[6]  Li Dong,et al.  Adaptive downsampling to improve image compression at low bit rates , 2006, IEEE Transactions on Image Processing.

[7]  Kwang In Kim,et al.  Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Yoshua Bengio,et al.  NICE: Non-linear Independent Components Estimation , 2014, ICLR.

[9]  Markus H. Gross,et al.  Perceptually based downscaling of images , 2015, ACM Trans. Graph..

[10]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[11]  Max Welling,et al.  Sylvester Normalizing Flows for Variational Inference , 2018, UAI.

[12]  Ullrich Köthe,et al.  Guided Image Generation with Conditional Invertible Neural Networks , 2019, ArXiv.

[13]  David Minnen,et al.  Joint Autoregressive and Hierarchical Priors for Learned Image Compression , 2018, NeurIPS.

[14]  Michael Elad,et al.  Down-Scaling for Better Transform Compression , 2001, Scale-Space.

[15]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[16]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[18]  Arnold W. M. Smeulders,et al.  i-RevNet: Deep Invertible Networks , 2018, ICLR.

[19]  C.E. Shannon,et al.  Communication in the Presence of Noise , 1949, Proceedings of the IRE.

[20]  David Duvenaud,et al.  Residual Flows for Invertible Generative Modeling , 2019, NeurIPS.

[21]  Arun N. Netravali,et al.  Reconstruction filters in computer-graphics , 1988, SIGGRAPH.

[22]  John D. Fernandez,et al.  Facial feature detection using Haar classifiers , 2006 .

[23]  Léon Bottou,et al.  Towards Principled Methods for Training Generative Adversarial Networks , 2017, ICLR.

[24]  Luc Van Gool,et al.  Generative Adversarial Networks for Extreme Learned Image Compression , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[26]  Rainer Lienhart,et al.  An extended set of Haar-like features for rapid object detection , 2002, Proceedings. International Conference on Image Processing.

[27]  Pieter Peers,et al.  Content-adaptive image downscaling , 2013, ACM Trans. Graph..

[28]  Jinwoo Shin,et al.  Neural Adaptive Content-aware Internet Video Delivery , 2018, OSDI.

[29]  Sergey Levine,et al.  VideoFlow: A Flow-Based Generative Model for Video , 2019, ArXiv.

[30]  David Duvenaud,et al.  Invertible Residual Networks , 2018, ICML.

[31]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[32]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[33]  Valero Laparra,et al.  End-to-end Optimized Image Compression , 2016, ICLR.

[34]  Michael Goesele,et al.  Rapid, detail-preserving image downscaling , 2016, ACM Trans. Graph..

[35]  Daniel E. Worrall,et al.  Reversible GANs for Memory-Efficient Image-To-Image Translation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Aline Roumy,et al.  Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding , 2012, BMVC.

[37]  Junjie Liu,et al.  $L_{0}$ -Regularized Image Downscaling , 2018, IEEE Transactions on Image Processing.

[38]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[40]  Richard H. Sherman,et al.  Chaotic communications in the presence of noise , 1993, Optics & Photonics.

[41]  Houqiang Li,et al.  Learning a Convolutional Neural Network for Image Compact-Resolution , 2019, IEEE Transactions on Image Processing.

[42]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[43]  Raanan Fattal,et al.  Image and video upscaling from local self-examples , 2011, TOGS.

[44]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[46]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[47]  Zhenzhong Chen,et al.  Learned Image Downscaling for Upscaling Using Content Adaptive Resampler , 2019, IEEE Transactions on Image Processing.

[48]  Kyoung Mu Lee,et al.  Task-Aware Image Downscaling , 2018, ECCV.

[49]  Ci Wang,et al.  Down-Sampling Based Video Coding Using Super-Resolution Technique , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[50]  Narendra Ahuja,et al.  Single image super-resolution from transformed self-exemplars , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Lubomir D. Bourdev,et al.  Real-Time Adaptive Image Compression , 2017, ICML.

[52]  N NetravaliArun,et al.  Reconstruction filters in computer-graphics , 1988 .

[53]  Chao Zhang,et al.  Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution , 2018, NeurIPS.

[54]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[55]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[56]  Proceedings of the 16th ACM Workshop on Hot Topics in Networks , 2017, ACM Workshop on Hot Topics in Networks.

[57]  Yoshua Bengio,et al.  Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation , 2013, ArXiv.

[58]  Horst Bischof,et al.  Fast and accurate image upscaling with super-resolution forests , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Prafulla Dhariwal,et al.  Glow: Generative Flow with Invertible 1x1 Convolutions , 2018, NeurIPS.

[60]  Samy Bengio,et al.  Density estimation using Real NVP , 2016, ICLR.

[61]  Shu-Tao Xia,et al.  Second-Order Attention Network for Single Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[63]  Changsheng Li,et al.  Residual Invertible Spatio-Temporal Network for Video Super-Resolution , 2019, AAAI.

[64]  David Minnen,et al.  Variational image compression with a scale hyperprior , 2018, ICLR.

[65]  David Duvenaud,et al.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models , 2018, ICLR.

[66]  Xiangjun Zhang,et al.  Low Bit-Rate Image Compression via Adaptive Down-Sampling and Constrained Least Squares Upconversion , 2009, IEEE Transactions on Image Processing.