Lucas-Kanade Reloaded: End-to-End Super-Resolution from Raw Image Bursts

This presentation addresses the problem of reconstructing a high-resolution image from multiple lower-resolution snapshots captured from slightly different viewpoints in space and time. Key challenges for solving this superresolution problem include (i) aligning the input pictures with sub-pixel accuracy, (ii) handling raw (noisy) images for maximal faithfulness to native camera data, and (iii) designing/learning an image prior (regularizer) well suited to the task. We address these three challenges with a hybrid algorithm building on the insight from [45] that aliasing is an ally in this setting, with parameters that can be learned end to end, while retaining the interpretability of classical approaches to inverse problems. The effectiveness of our approach is demonstrated on synthetic and real image bursts, setting a new state of the art on several benchmarks and delivering excellent qualitative results on real raw bursts captured by smartphones and prosumer cameras. Our code is available at https://github.com/ bruno-31/lkburst.git.

[1]  Gabriele Facciolo,et al.  Joint Demosaicking and Denoising by Fine-Tuning of Bursts of Raw Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[2]  Ron Kimmel,et al.  Demosaicing: Image Reconstruction from Color CCD Samples , 1998, ECCV.

[3]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[4]  Li Chen,et al.  A Nonlinear Least Square Technique for Simultaneous Image Registration and Super-Resolution , 2007, IEEE Transactions on Image Processing.

[5]  Hanseok Ko,et al.  NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Xiaohan Chen,et al.  Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds , 2018, NeurIPS.

[7]  Wenbo Li,et al.  MuCAN: Multi-Correspondence Aggregation Network for Video Super-Resolution , 2020, ECCV.

[8]  Russell C. Hardie,et al.  A Fast Image Super-Resolution Algorithm Using an Adaptive Wiener Filter , 2007, IEEE Transactions on Image Processing.

[9]  Luc Van Gool,et al.  Replacing Mobile Camera ISP with a Single Deep Learning Model , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[10]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Thomas S. Huang,et al.  Multiframe image restoration and registration , 1984 .

[12]  J. Ponce,et al.  Fully Trainable and Interpretable Non-local Sparse Models for Image Restoration , 2019, ECCV.

[13]  Luc Van Gool,et al.  Deep Unfolding Network for Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[15]  Donald Geman,et al.  Nonlinear image recovery with half-quadratic regularization , 1995, IEEE Trans. Image Process..

[16]  Cosmin Bercea,et al.  Confidence-aware Levenberg-Marquardt optimization for joint motion estimation and super-resolution , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[17]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[19]  Chen Change Loy,et al.  EDVR: Video Restoration With Enhanced Deformable Convolutional Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[20]  Michaël Gharbi,et al.  Basis Prediction Networks for Effective Burst Denoising With Large Kernels , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Seoung Wug Oh,et al.  Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Enrico Magli,et al.  DeepSUM: Deep Neural Network for Super-Resolution of Unregistered Multitemporal Images , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Stanley H. Chan,et al.  Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications , 2016, IEEE Transactions on Computational Imaging.

[25]  Marc Levoy,et al.  Handheld multi-frame super-resolution , 2019, ACM Trans. Graph..

[26]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[27]  M. Vetterli,et al.  Super-resolution from Unregistered Aliased Images , 2006 .

[28]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[29]  Bruce H. Pillman.  Super-Resolution Imaging , 2013, J. Electronic Imaging.

[30]  Hsieh Hou,et al.  Cubic splines for image interpolation and digital filtering , 1978 .

[31]  Xiaohan Chen,et al.  Plug-and-Play Methods Provably Converge with Properly Trained Denoisers , 2019, ICML.

[32]  Russell C. Hardie,et al.  Joint MAP registration and high-resolution image estimation using a sequence of undersampled images , 1997, IEEE Trans. Image Process..

[33]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[34]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[35]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[36]  Brendt Wohlberg,et al.  Plug-and-Play priors for model based reconstruction , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[37]  Peyman Milanfar,et al.  Kernel Regression for Image Processing and Reconstruction , 2007, IEEE Transactions on Image Processing.

[38]  Michael Elad,et al.  Advances and challenges in super‐resolution , 2004, Int. J. Imaging Syst. Technol..

[39]  Jonathan T. Barron,et al.  Unprocessing Images for Learned Raw Denoising , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Joos Vandewalle,et al.  Aliasing is Good for You: Joint Registration and Reconstruction for Super-Resolution , 2006 .

[41]  Radu Timofte,et al.  NTIRE 2021 Challenge on Burst Super-Resolution: Methods and Results , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[42]  Luc Van Gool,et al.  Deep Burst Super-Resolution , 2021 .

[43]  Javier Sánchez,et al.  The Inverse Compositional Algorithm for Parametric Registration , 2016, Image Process. Line.

[44]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[45]  Thomas Kohler,et al.  Multi-Frame Super-Resolution Reconstruction with Applications to Medical Imaging , 2018, 1812.09375.

[46]  Michael Elad,et al.  Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images , 1997, IEEE Trans. Image Process..

[47]  Thomas S. Huang,et al.  Image Super-Resolution: Historical Overview and Future Challenges , 2017 .

[48]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Yoshua Bengio,et al.  HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery , 2020, ArXiv.

[50]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.