Real-Time Under-Display Cameras Image Restoration and HDR on Mobile Devices

The new trend of full-screen devices implies positioning the camera behind the screen to bring a larger display-to-body ratio, enhance eye contact, and provide a notch-free viewing experience on smartphones, TV or tablets. On the other hand, the images captured by under-display cameras (UDCs) are degraded by the screen in front of them. Deep learning methods for image restoration can significantly reduce the degradation of captured images, providing satisfying results for the human eyes. However, most proposed solutions are unreliable or efficient enough to be used in real-time on mobile devices. In this paper, we aim to solve this image restoration problem using efficient deep learning methods capable of processing FHD images in real-time on commercial smartphones while providing high-quality results. We propose a lightweight model for blind UDC Image Restoration and HDR, and we also provide a benchmark comparing the performance and runtime of different methods on smartphones. Our models are competitive on UDC benchmarks while using x4 less operations than others. To the best of our knowledge, we are the first work to approach and analyze this real-world single image restoration problem from the efficiency and production point of view.

[1]  Marcos V. Conde,et al.  Perceptual Image Enhancement for Smartphone Real-Time Applications , 2022, 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[2]  Chen Change Loy,et al.  MIPI 2022 Challenge on Under-Display Camera Image Restoration: Methods and Results , 2022, ECCV Workshops.

[3]  Tao Wang,et al.  Under-Display Camera Image Enhancement via Cascaded Curve Estimation , 2022, IEEE Transactions on Image Processing.

[4]  Jaihyun Koh,et al.  BNUDC: A Two-Branched Deep Neural Network for Restoring Images from Under-Display Cameras , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Jimmy S. J. Ren,et al.  NTIRE 2022 Challenge on Perceptual Image Quality Assessment , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Marcos V. Conde,et al.  Conformer and Blind Noisy Students for Improved Image Quality Assessment , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Steven G. McDonagh,et al.  Model-Based Image Signal Processors via Learnable Dictionaries , 2022, AAAI.

[8]  A. Leonardis,et al.  FlexHDR: Modeling Alignment and Exposure Uncertainties for Flexible HDR Imaging , 2022, IEEE Transactions on Image Processing.

[9]  Su-Jin Lee,et al.  Controllable Image Restoration for Under-Display Camera in Smartphones , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Radu Timofte,et al.  NTIRE 2021 Challenge on Image Deblurring , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[11]  Jinwei Gu,et al.  Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Neil Emerton,et al.  Image Restoration for Under-Display Camera , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Lai-Man Po,et al.  UDC 2020 Challenge on Image Restoration of Under-Display Camera: Methods and Results , 2020, ECCV Workshops.

[14]  Liu Guanghui,et al.  28‐2: Study of the Image Blur through FFS LCD Panel Caused by Diffraction for Camera under Panel , 2020 .

[15]  Yung-Yu Chuang,et al.  Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  J. Giovannelli,et al.  Bayesian estimation of regularization and point spread function parameters for Wiener-Hunt deconvolution , 2020 .

[17]  Yifan Peng,et al.  Learned large field-of-view imaging with thin-plate optics , 2019, ACM Trans. Graph..

[18]  Sabine Süsstrunk,et al.  Kernel Modeling Super-Resolution on Real Low-Resolution Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  M. Irani,et al.  Blind Super-Resolution Kernel Estimation using an Internal-GAN , 2019, NeurIPS.

[20]  Wangmeng Zuo,et al.  Blind Super-Resolution With Iterative Kernel Correction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Ke Wang,et al.  AI Benchmark: Running Deep Neural Networks on Android Smartphones , 2018, ECCV Workshops.

[22]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[23]  Jason Yosinski,et al.  An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution , 2018, NeurIPS.

[24]  Siyuan Liu,et al.  Unsupervised Image Super-Resolution Using Cycle-in-Cycle Generative Adversarial Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[25]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Wangmeng Zuo,et al.  Learning a Single Convolutional Super-Resolution Network for Multiple Degradations , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[28]  Aswin C. Sankaranarayanan,et al.  FlatCam: Thin, Lensless Cameras Using Coded Aperture and Computation , 2017, IEEE Transactions on Computational Imaging.

[29]  Yi-Pai Huang,et al.  Evaluation of a Transparent Display's Pixel Structure Regarding Subjective Quality of Diffracted See-Through Images , 2017, IEEE Photonics Journal.

[30]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[31]  Rynson W. H. Lau,et al.  Learning Fully Convolutional Networks for Iterative Non-blind Deconvolution , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Yi-Pai Huang,et al.  See-Through Image Blurring of Transparent Organic Light-Emitting Diodes Display: Calculation Method Based on Diffraction and Analysis of Pixel Structures , 2016, Journal of Display Technology.

[33]  Chang-Mo Yang,et al.  Modeling of Luminance Transition Curve of Transparent Plastics on Transparent OLED Displays , 2016, Color Imaging: Displaying, Processing, Hardcopy, and Applications.

[34]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[35]  Ce Liu,et al.  Deep Convolutional Neural Network for Image Deconvolution , 2014, NIPS.

[36]  Bernhard Schölkopf,et al.  A Machine Learning Approach for Non-blind Image Deconvolution , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Chang-Yeong Kim,et al.  50.2: Adding Depth‐Sensing Capability to an OLED Display System Based on Coded Aperture Imaging , 2013 .

[38]  Chang-Yeong Kim,et al.  P‐135: An LCD Display System with Depth‐Sensing Capability Based on Coded Aperture Imaging , 2012 .

[39]  Seungyong Lee,et al.  Handling outliers in non-blind image deconvolution , 2011, 2011 International Conference on Computer Vision.

[40]  Andrew Zisserman,et al.  Deblurring Shaken and Partially Saturated Images , 2011, International Journal of Computer Vision.

[41]  Frédo Durand,et al.  Understanding and evaluating blind deconvolution algorithms , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Richard Szeliski,et al.  PSF estimation using sharp edge prediction , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.