Joint Demosaicking and Denoising in the Wild: The Case of Training Under Ground Truth Uncertainty

Image demosaicking and denoising are the two key fundamental steps in digital camera pipelines, aiming to reconstruct clean color images from noisy luminance readings. In this paper, we propose and study Wild-JDD, a novel learning framework for joint demosaicking and denoising in the wild. In contrast to previous works which generally assume the ground truth of training data is a perfect reflection of the reality, we consider here the more common imperfect case of ground truth uncertainty in the wild. We first illustrate its manifestation as various kinds of artifacts including zipper effect, color moire and residual noise. Then we formulate a two-stage data degradation process to capture such ground truth uncertainty, where a conjugate prior distribution is imposed upon a base distribution. After that, we derive an evidence lower bound (ELBO) loss to train a neural network that approximates the parameters of the conjugate prior distribution conditioned on the degraded input. Finally, to further enhance the performance for out-of-distribution input, we design a simple but effective fine-tuning strategy by taking the input as a weakly informative prior. Taking into account ground truth uncertainty, Wild-JDD enjoys good interpretability during optimization. Extensive experiments validate that it outperforms state-of-the-art schemes on joint demosaicking and denoising tasks on both synthetic and realistic raw datasets. Introduction Modern digital cameras use a single sensor overlaid with a color filter array (CFA) to capture an image. This means that only one color channel’s value is recorded for each pixel location. LetN be the number of pixels in an image, the raw data acquisition process can be simply modeled as x = Az + n, (1) where x ∈ R is a noisy raw data vector of luminance readings, A ∈ RN×3N is a mosaicking operation, z ∈ R is an unknown clean image with three color channels, and n ∈ R is a noise vector. Before the final “cooked” image is ready for the users, the raw data undergoes a series of processing steps, known as the image processing pipeline. Among those, demosaicking and denoising (DM&DN) are two of the very early and Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. (a) Zipper effect. (b) Color moire. (c) Residual noise. Figure 1: Imperfect ground truth examples (electronic zoomin recommended): (a) A ground truth image from CBSD dataset (Arbeláez et al. 2011) suffering from zipper effect, an artificial jagged pattern around edges; (b) Color moire in an image from ImageNet dataset (Russakovsky et al. 2015). Such artifact appears as false coloring due to interpolation error; (c) Noticeable residual noise in the collected “clean” image from Renoir dataset (Anaya and Barbu 2018). most crucial steps. Demosaicking aims to undo the mosaicking operation A by interpolating the missing two-thirds of each pixel’s color channels, while denoising removes the inevitable noise n from the measurement x. Due to their modular property, substantial traditional literature takes them as independent tasks and executes them in a sequential manner. This yields potentially suboptimal performance, and inspires several works on jointly addressing the DM&DN tasks (Liu et al. 2020; Kokkinos and Lefkimmiatis 2019; Tan et al. 2017a). Among the joint DM&DN works, data-driven approaches (Liu et al. 2020; Tan et al. 2018; Kokkinos and Lefkimmiatis 2018) have been shown more effective than applying handcrafted priors and filters. These approaches usually require a collection of paired data, which are the mosaicked noisy images x and the demosaicked clean “ground truth” counterparts y. However, it is often costly and tedious to collect a large amount of high quality real-life data. Furthermore, the collected y is not perfect without artifacts or noise. We illustrate this in Figure 1. For demosaicking, many approaches (Syu, Chen, and Chuang 2018; Tan et al. 2017b) take the output from a camera pipeline as y, possibly introducing artifacts like zipper effect or color moire in regions with rich textures and sharp edges. For denoising, the ar X iv :2 10 1. 04 44 2v 1 [ cs .C V ] 1 2 Ja n 20 21 “clean” images are often collected by either setting a lowISO (Plotz and Roth 2017; Anaya and Barbu 2018) or averaging a set of repeated shots of the same scene (Abdelhamed, Lin, and Brown 2018), which still contain noticeable noise. Moreover, such denoising data collecting process usually assumes the captured objects to be perfectly still, or requires a precise spatial alignment and intensity calibration among a burst of images. Potential failure cases would introduce additional error into the collected dataset. Therefore, all these in-the-wild issues means that the “ground truth” y deviates from the needed authentic z, limiting the performance of DM&DN model. To account for the fact that the collected ground truth y is not a perfect reflection of z, we propose Wild-JDD, a novel joint demosaiking and denoising learning framework to enable training under ground truth uncertainty. In WildJDD, we first formulate a two-stage data degradation process, where a conjugate prior distribution is imposed upon a base Gaussian distribution. Then, we derive an ELBO loss from a variational perspective. In this way, the optimization process is aware of the target uncertainty and prevents the trained neural network from over-fitting to those randomness errors. Beyond that, when the testing image falls outside of the training range, we further enhance the performance by regarding the input as a weakly informative prior. Our main contributions are summarized as follows: • We identify in existing DM&DN datasets the ground truth uncertainty issues, manifesting themselves as various artifacts in the wild, such as zipper effect, color moire and residual noise. • We introduce a novel learning framework for joint demosaicking and denoising in the wild (Wild-JDD), where a two-stage data degradation and an ELBO loss are formulated for optimization. We also propose a simple but effective fine-tuning strategy for out-of-distribution input. • Instead of simply generating a demosaicked clean image, networks instantiated from our framework are capable of estimating all the parameters involved in data degradation and reconstruction, which provides better interpretability of the optimization process. • We conduct extensive experiments on both synthetic and realistic datasets. Quantitative and qualitative comparisons show that Wild-JDD substantially outperforms state-of-the-art works.