Image Restoration for Under-Display Camera

The new trend of full-screen devices encourages us to position a camera behind a screen. Removing the bezel and centralizing the camera under the screen brings larger display-to-body ratio and enhances eye contact in video chat, but also causes image degradation. In this paper, we focus on a newly-defined Under-Display Camera (UDC), as a novel real-world single image restoration problem. First, we take a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED) and analyze their optical systems to understand the degradation. Second, we design a novel Monitor-Camera Imaging System (MCIS) for easier real pair data acquisition, and a model-based data synthesizing pipeline to generate UDC data only from display pattern and camera measurements. Finally, we resolve the complicated degradation using learning-based methods. Our model demonstrates a real-time high-quality restoration trained with either real or synthetic data. The presented results and methods provide good practice to apply image restoration to real-world applications.

[1]  Zhenhua Zhang.  Image Deblurring of Camera Under Display by Deep Learning , 2020 .

[2]  Laura Waller,et al.  Learned reconstructions for practical mask-based lensless imaging , 2019, Optics express.

[3]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Stefan Roth,et al.  Benchmarking Denoising Algorithms with Real Photographs , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Qin Xu,et al.  Learning Raw Image Denoising With Bayer Pattern Unification and Bayer Preserving Augmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Radu Timofte,et al.  Dense-Haze: A Benchmark for Image Dehazing with Dense-Haze and Haze-Free Images , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[8]  Michael T. Heath,et al.  Scientific Computing: An Introductory Survey , 1996 .

[9]  Lei Zhang,et al.  FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising , 2017, IEEE Transactions on Image Processing.

[10]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  J. Goodman.  Introduction to Fourier optics , 1969 .

[12]  Kaushik Mitra,et al.  Towards Photorealistic Reconstruction of Highly Multiplexed Lensless Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[13]  Zhiwei Xiong,et al.  Camera Lens Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Jonathan T. Barron,et al.  Unprocessing Images for Learned Raw Denoising , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Samuel W. Hasinoff,et al.  Photon, Poisson Noise , 2014, Computer Vision, A Reference Guide.

[16]  Vishal M. Patel,et al.  Image De-Raining Using a Conditional Generative Adversarial Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Jia Xu,et al.  Learning to See in the Dark , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Lai-Man Po,et al.  UDC 2020 Challenge on Image Restoration of Under-Display Camera: Methods and Results , 2020, ECCV Workshops.

[20]  Richard G. Baraniuk,et al.  Face Detection and Verification Using Lensless Cameras , 2019, IEEE Transactions on Computational Imaging.

[21]  Fahad Shahbaz Khan,et al.  NTIRE 2019 Challenge on Video Deblurring: Methods and Results , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Thomas Huang,et al.  Adaptation Strategies for Applying AWGN-Based Denoiser to Realistic Noise , 2019, AAAI.

[23]  Wolfgang Heidrich,et al.  Learning Rank-1 Diffractive Optics for Single-Shot High Dynamic Range Imaging , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yifan Peng,et al.  Learned large field-of-view imaging with thin-plate optics , 2019, ACM Trans. Graph..

[25]  Ling-Yu Duan,et al.  Benchmarking Single-Image Reflection Removal Algorithms , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[26]  Louis L. Scharf,et al.  A Multistage Representation of the Wiener Filter Based on Orthogonal Projections , 1998, IEEE Trans. Inf. Theory.

[27]  Jonathan T. Barron,et al.  Learning to Synthesize Motion Blur , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[29]  池内 克史,et al.  Computer Vision: A Reference Guide , 2014 .

[30]  Xiaochun Cao,et al.  Single Image Dehazing via Multi-scale Convolutional Neural Networks , 2016, ECCV.

[31]  Adrian Barbu,et al.  RENOIR - A dataset for real low-light image noise reduction , 2014, J. Vis. Commun. Image Represent..

[32]  Stephen Lin,et al.  A High-Quality Denoising Dataset for Smartphone Cameras , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Michael S. Brown,et al.  Reflection Removal Using a Dual-Pixel Sensor , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Chin‐Ti Chen,et al.  Organic Light-Emitting Diode (OLED) , 2014 .

[35]  J. Giovannelli,et al.  Bayesian estimation of regularization and point spread function parameters for Wiener-Hunt deconvolution , 2020 .

[36]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[37]  Yang Wang,et al.  When AWGN-based Denoiser Meets Real Noises , 2019, AAAI.

[38]  Vishal M. Patel,et al.  Density-Aware Single Image De-raining Using a Multi-stream Dense Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Dong-Wook Kim,et al.  NTIRE 2019 Challenge on Real Image Denoising: Methods and Results , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[40]  Wangmeng Zuo,et al.  Toward Convolutional Blind Denoising of Real Photographs , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[42]  Richard Szeliski,et al.  Automatic Estimation and Removal of Noise from a Single Image , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Raanan Fattal,et al.  Single image dehazing , 2008, ACM Trans. Graph..

[44]  Vladlen Koltun,et al.  Zoom to Learn, Learn to Zoom , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Michael S. Brown,et al.  Noise Flow: Noise Modeling With Conditional Normalizing Flows , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[46]  Neil Emerton,et al.  Aperture Design for Learning-based Image Restoration , 2020 .

[47]  Neil Emerton,et al.  74‐1: Image Restoration for Display‐Integrated Camera , 2020 .

[48]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.