Lensless Imaging with Focusing Sparse URA Masks in Long-Wave Infrared and Its Application for Human Detection

We introduce a lensless imaging framework for contemporary computer vision applications in long-wavelength infrared (LWIR). The framework consists of two parts: a novel lensless imaging method that utilizes the idea of local directional focusing for optimal binary sparse coding, and lensless imaging simulator based on Fresnel-Kirchhoff diffraction approximation. Our lensless imaging approach, besides being computationally efficient, is calibration-free and allows for wide FOV imaging. We employ our lensless imaging simulation software for optimizing reconstruction parameters and for synthetic image generation for CNN training. We demonstrate the advantages of our framework on a dual-camera system (RGB-LWIR lensless), where we perform CNNbased human detection using the fused RGB-LWIR data.

[1]  Nicu Sebe,et al.  Learning Cross-Modal Deep Representations for Robust Pedestrian Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  J. Tanida,et al.  Thin Observation Module by Bound Optics (TOMBO): Concept and Experimental Verification. , 2001, Applied optics.

[3]  Namil Kim,et al.  Multispectral pedestrian detection: Benchmark dataset and baseline , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  E. E. Fenimore,et al.  Uniformly redundant arrays , 1977 .

[5]  Q. Shen,et al.  Hard x-ray microscopy with Fresnel zone plates reaches 40 nm Rayleigh resolution. , 2008 .

[6]  R. Fergus,et al.  Random Lens Imaging , 2006 .

[7]  Aswin C. Sankaranarayanan,et al.  FlatCam: Thin, Lensless Cameras Using Coded Aperture and Computation , 2017, IEEE Transactions on Computational Imaging.

[8]  H. Barrett.  Fresnel zone plate imaging in nuclear medicine. , 1972, Journal of nuclear medicine : official publication, Society of Nuclear Medicine.

[9]  Ashok Veeraraghavan,et al.  PhaseCam3D — Learning Phase Masks for Passive Single View Depth Estimation , 2019, 2019 IEEE International Conference on Computational Photography (ICCP).

[10]  Dragoljub Pokrajac,et al.  People detection in low resolution infrared videos , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[11]  Shu Wang,et al.  Multispectral Deep Neural Networks for Pedestrian Detection , 2016, BMVC.

[12]  Ayan Chakrabarti,et al.  Learning Sensor Multiplexing Design through Back-propagation , 2016, NIPS.

[13]  Tatsuya Harada,et al.  Multispectral Object Detection for Autonomous Vehicles , 2017, ACM Multimedia.

[14]  Stephen P. Boyd,et al.  End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging , 2018, ACM Trans. Graph..

[15]  Angel Domingo Sappa,et al.  Multimodal Stereo Vision System: 3D Data Extraction and Algorithm Evaluation , 2012, IEEE Journal of Selected Topics in Signal Processing.

[16]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Bernard Ghanem,et al.  End-to-end Learned, Optically Coded Super-resolution SPAD Camera , 2020, ACM Trans. Graph..

[18]  Heiko Neumann,et al.  Fully Convolutional Region Proposal Networks for Multispectral Person Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[19]  Tatiana Grulois,et al.  Extra-thin infrared camera for low-cost surveillance applications. , 2014, Optics letters.

[20]  E. E. Fenimore,et al.  Uniformly redundant arrays: digital reconstruction methods. , 1981, Applied optics.

[21]  Yifan Peng,et al.  Deep Optics for Single-Shot High-Dynamic-Range Imaging , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Francois P. J. le Roux,et al.  Non-uniformity correction and bad pixel replacement on LWIR and MWIR images , 2011, 2011 Saudi International Electronics, Communications and Photonics Conference (SIECPC).

[23]  Alistair A. Young,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2017, MICCAI 2017.

[24]  Michael J. DeWeert,et al.  Lensless coded aperture imaging with separable doubly Toeplitz masks , 2014, Sensing Technologies + Applications.

[25]  Robert Pless,et al.  SparkleGeometry: Glitter Imaging for 3D Point Tracking , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Laura Waller,et al.  DiffuserCam: Lensless Single-exposure 3D Imaging , 2017, ArXiv.

[27]  Janos Kirz,et al.  Phase zone plates for x rays and the extreme uv , 1974 .

[28]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Patrick R. Gill,et al.  Thermal Escher sensors: Pixel-efficient lensless imagers based on tiled optics , 2017 .

[31]  Gordon Wetzstein,et al.  Deep Optics for Monocular Depth Estimation and 3D Object Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Sven Behnke,et al.  Multispectral Pedestrian Detection using Deep Fusion Convolutional Neural Networks , 2016, ESANN.

[33]  T. M. Cannon,et al.  Coded aperture imaging with uniformly redundant arrays. , 1978, Applied optics.

[34]  Rajesh Menon,et al.  Lensless Photography with only an image sensor , 2017, Applied optics.

[35]  Zhou Wang,et al.  Translation insensitive image similarity in complex wavelet domain , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[36]  Thomas B. Moeslund,et al.  Thermal cameras and applications: a survey , 2013, Machine Vision and Applications.

[37]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[38]  M. Schmid.  Principles Of Optics Electromagnetic Theory Of Propagation Interference And Diffraction Of Light , 2016 .

[39]  Kaushik Mitra,et al.  Towards Photorealistic Reconstruction of Highly Multiplexed Lensless Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[40]  Bernt Schiele,et al.  Ten Years of Pedestrian Detection, What Have We Learned? , 2014, ECCV Workshops.

[41]  Kelum A. A. Gamage,et al.  Coded-aperture imaging systems: Past, present and future development - A review , 2016 .

[42]  Riad I. Hammoud,et al.  Thermal-Visible Video Fusion for Moving Target Tracking and Pedestrian Classification , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Jean J. M. in't Zand,et al.  A coded-mask imager as monitor of Galactic X-ray sources , 1992 .