Single-Sensor RGB and NIR Image Acquisition: Toward Optimal Performance by Taking Account of CFA Pattern, Demosaicking, and Color Correction

In recent years, many applications using a pair of RGB and near-infrared (NIR) images have been proposed in computer vision and image processing communities. Thanks to recent progress of image sensor technology, it is also becoming possible to manufacture an image sensor with a novel spectral filter array, which has RGB plus NIR pixels for one-shot acquisition of the RGB and the NIR images. In such a novel filter array, half of the G pixels in the standard Bayer color filter array (CFA) are typically replaced with the NIR pixels. However, its performance has not fully been investigated in the pipeline of single-sensor RGB and NIR image acquisition. In this paper, we present an imaging pipeline of the single-sensor RGB and NIR image acquisition and investigate its optimal performance by taking account of the filter array pattern, demosaicking and color correction. We also propose two types of filter array patterns and demosaicking algorithms for improving the quality of acquired RGB and NIR images. Based on the imaging pipeline we present, the performance of different filter array patterns and demosaicking algorithms is evaluated. In experimental results, we demonstrate that our proposed filter array patterns and demosaicking algorithms outperform the existing ones. Introduction In recent years, many applications using a pair of RGB and near-infrared (NIR) images have been proposed in computer vision and image processing communities such as image enhancement [1, 2], image fusion [3, 4], dehazing [5, 6], denoising [7, 8], and shadow detection [9]. However, the acquisition of the pair of RGB and NIR images is still a challenging task because existing acquisition systems typically require multiple cameras [1] or multiple shots [9], where one is required for RGB and the other is required for NIR. In current compact and low-cost digital cameras, singlesenor color image acquisition with the Bayer color filter array (CFA) [10], as shown in Fig. 1 (a)-(c), is well established [11]. To simultaneously acquire the RGB and the NIR images, many existing works extend the idea of using the CFA for single-sensor RGB and NIR image acquisition [12–18]. Thanks to recent progress of image sensor technology, it is also becoming possible to manufacture an image sensor with a novel filter array, which has RGB plus NIR pixels [14–18]. This sensor can provide us with a practical solution for one-shot acquisition of the RGB and the NIR images without increased size and cost from current color digital cameras. Hereafter, we call such a filter array “RGB-NIR filter 0 5 10 15 20 25 30 35 40 400 500 600 700 800 90