Non-negative Matrix Completion for the Enhancement of Snapshot Mosaic Multispectral Imagery

Multiand Hyperspectral Imaging (HSI) are characterized by the discrepancy between the dimensionality of hyperspectral image and video data and the dimensionality of the spectral detectors. This issue has been addressed by various architectures, including the Snapshot Mosaic Multispectral Imaging architecture, where each pixel (or group of pixels) is associated with a single spectral band. An unavoidable side effect of this design is the hard trade-off between the spatial and the spectral resolution. In this work, we propose a formal approach for overcoming this trade-off by formulating the problem of full resolution recovery as a low rank Matrix Completion problem. Furthermore, we extend the traditional formulation of Matrix Completion by introducing non-negativity constraints during the recovery process, thus significantly enhancing the reconstruction quality. Experimental results suggest that the Non-Negative Matrix Completion (NN-MC) framework is capable of estimating a high spatial and spectral resolution hypercube from a single exposure, surpassing state-of-the-art schemes like nearest-neighbors as well as the unconstrained Matrix Completion technique. Introduction A fundamental issue that hyperspectral imaging sensors have to address is how to collect the four dimensional HSI data, two spatial, one spectral, and one temporal, using a single detector, an 1D array, or 2D plane detectors. The discrepancy between the requested and the available dimensionality of detectors has sparked different philosophies in hyperspectral image acquisition designs, leading to spatial, spectral, and frame scanning architectures [1]. A shortcoming shared by these approaches is the scanning requirements for constructing the complete 3D hyperspectral datacube, where in the case of spatial/spectral scanning, multiple lines/pixels have to be scanned, while for 2D frame scanning systems, multiple frames have to be acquired in order to obtain the complete spectral profile of the scene [2]. These limitations are responsible for a number of issues that hinder the HSI performance, including slow acquisition time and motion artifacts. Furthermore, the need for miniaturization of the imaging systems implies that novel designs should strive to be free of mechanical parts, such as moving mirrors, since they limit the temporal resolution and increase the complexity of the designs. Recent approaches address these limitations by employing novel hardware and sophisticated signal processing techniques to achieve similar performance and imaging capabilities. Snapshot (or Simultaneous) Spectral Imaging (SSI) systems acquire the complete spatio-spectral cube from a single or a few captured frames, i.e., during a single or a few integration periods, without the need for successive frame acquisition [3]. While earlier approaches relied on additional hardware, such as coherent fiber bundles and mirror slicers to satisfy the requirements for SSI, more recent paradigms employ novel light manipulation components and state-of-the-art signal processing to achieve this task. One such prominent case is the Snapshot Mosaic Multispectral Imaging architectures, also known as hyper/multi-spectral Color Filter Arrays. This paradigm relies on the use of Spectrally Resolved Detector Arrays (SRDA) where each pixel is associated with a specific spectral region, thus allowing the acquisition of a full hyperspectral cubes from a single exposure [4, 5]. Unfortunately, to achieve high temporal resolution imaging, SRDA architectures must sacrifice spatial resolution since only a small subset of pixels acquire images from a specific spectral band. In practice, pixel binning is performed where groups of spectral-specific pixels are grouped together in full spectral resolution super-pixels. The process is depicted in Figure 1. Figure 1: SRDA architecture (left), a snapshot mosaic raw frame (center), a full spectral resolution ”super-pixel” as part of the reconstruction hypercube(right). Notice that the process of producing the ”super-pixels” leads to dramatically smaller spatial resolution. Objectives and state-of-the-art The objective of this work is to provide a formal method for addressing the challenging spatio-spectral trade-off that characterizes Snapshot Mosaic Multispectral Imaging architecture relying on SRDA detectors. More specifically, SRDA detectors perform spatial subsampling of each spectral band by producing a two-dimensional array of “super-pixels” where each such “super-pixel” corresponds to a binned group of physical pixels, containing measurements from multiple spectral bands. As a consequence, the effective spatial resolution for each spectral band is given by the total number of pixels divided by the number of binned pixels in each super-pixel, as seen in Figure 1. According to our approach, to address this issue we exploit the inherent redundancies that exist in high dimensional hyperspectral data in order to accurately estimate the missing spectral bands from binned groups of pixels. Formally, the end goal is to generate a full spatial resolution hypercube M ∈ Rm×n×b + from a single exposure image M ∈ Rm×n + , thus allowing imaging of highly dynamic scenes or when imaging takes place in moving platforms such as UAV and satellites. The objective is visually depicted in Figure 2 where a raw snapshot, the corresponding lifted hypercube and the completed hypercube are illustrated. Figure 2: Illustration of recovery process. A raw snapshot (left) corresponds to a undersampled lifted hypercube (center) which must be completed to obtain the complete spectral profile of the scene (right). During the last decade, the concept of signal compressibility and sparsity has raised a lot of attention in the mathematics and signal processing communities which have treated novel concepts, including Compressed Sensing (CS) and Matrix Completion (MC), as part of a disruptive new framework which has revolutionized the way we efficiently sense, compress, and process visual information, e.g., [6, 7, 8, 9]. In a nutshell, according to the CS framework, a signal can be perfectly recovered from a severely under-sampled set of measurements provided the signal is sparse in some basis, and the basis on which the signal is sampled is incoherent with the basis on which the signal is sparse [10]. A prominent example of a CS based imaging architecture is the Single Pixel Camera (SPC), employing a Digital Micromirror Device to acquire scene information using a single detector element [11]. A Hyperspectral Single Pixel Camera corresponds to an extension of the typical SPC where the single detector element is replaced by a spectrometer [12]. The Compressive HS Imaging by Separable Spatial and Spectral operators (CHISSS) is an alternative spatial scanning architecture which employs a DMD placed before a grating which itself is modulated by a Coded Aperture before acquiring a two-dimensional measurement [13]. Similarly, the Coded Aperture Snapshot Spectral Imaging (CASSI) is a snapshot spectral imaging architecture which employs a DMD for spatially modulating the incoming light before it is dispersed by a grating and imaged by a 2D detector array [14]. More recently, a novel SSI architecture termed SpatialSpectral Encoded Compressive HS Imager (SSCSI) was proposed combining a spectral dispenser with a random shearing mask, extending the single wavelength computational light field acquisition architecture [15]. The CS framework has also been considered for the compression of multi and hyperspectral imaging without resorting to any modification in hardware, thus maintaining the limitations of current hyperspectral imagers [16]. Recently, the authors in [17] formulated a recovery method for SRDA HSI architectures based on a generalized inpainting approach, while a spatio-spectral Compressed Sensing based acquisition and recovery approach was proposed for HSI data acquisition [18]. Low Rank Matrix Completion Our approach is based on the recently proposed framework of Matrix Completion (MC) [19, 20] which has emerged as a disciplined way of addressing the recovery of high-dimensional data from what appears to be incomplete, and perhaps even corrupted information. Low rank MC has been utilized in a variety of image acquisition and processing tasks including the acquisition of High Dynamic Range Imaging [21] and video denoising [22] among others. More specifically, given a m× n measurement matrix M, recovering the (mn) entries of the matrix from a smaller number of k << mn entries is not possible, in general. However, it was recently shown that the recovery of the complete set of entries in a matrix is possible, provided that both the number of missing entries and the rank of the matrix are appropriately bounded. Formally, let A be a linear map from Rm×n → Rk, that selects a subset of the entries in the matrix M. The linear map A , is defined as a random sampling operator that records a small number of entries from the matrix M, that is A (mi j) = {1 if (i j) ∈ S | 0 otherwise}, where S is the sampling set. According to the low rank MC paradigm, we can estimate X from the undersampled matrix M, by solving: