Low-complexity image processing for a high-throughput low-latency snapshot multispectral imager with integrated tiled filters

Traditional spectral imaging cameras typically operate as pushbroom cameras by scanning a scene. This approach makes such cameras well-suited for high spatial and spectral resolution scanning applications, such as remote sensing and machine vision, but ill-suited for 2D scenes with free movement. This limitation can be overcome by single frame, multispectral (here called snapshot) acquisition, where an entire three-dimensional multispectral data cube is sensed at one discrete point in time and multiplexed on a 2D sensor. Our snapshot multispectral imager is based on optical filters monolithically integrated on CMOS image sensors with large layout flexibility. Using this flexibility, the filters are positioned on the sensor in a tiled layout, allowing trade-offs between spatial and spectral resolution. At system-level, the filter layout is complemented by an optical sub-system which duplicates the scene onto each filter tile. This optical sub-system and the tiled filter layout lead to a simple mapping of 3D spectral cube data on the sensor, facilitating simple cube assembly. Therefore, the required image processing consists of simple and highly parallelizable algorithms for reflectance and cube assembly, enabling real-time acquisition of dynamic 2D scenes at low latencies. Moreover, through the use of monolithically integrated optical filters the multispectral imager achieves the qualities of compactness, low cost and high acquisition speed, further differentiating it from other snapshot spectral cameras. Our prototype camera can acquire multispectral image cubes of 256x256 pixels over 32 bands in the spectral range of 600-1000nm at 340 cubes per second for normal illumination levels.