Temporal ghost Fourier compressive inference camera

The need for real-time processing fast moving objects in machine vision requires the cooperation of high frame rate camera and a large amount of computing resources. The cost, high detection bandwidth requirements, data and computation burden limit the wide applications of high frame rate machine vision. Compressive Video Sensing (CVS) allows capturing events at much higher frame rate with the slow camera, by reconstructing a frame sequence from a coded single image. At the same time, complex frame sequence reconstruction algorithms in CVS pose challenges for computing resources. Even though the reconstruction process is low computational complexity, image-dependent machine vision algorithms also suffers from a large amount of computing energy consumption. Here we present a novel CVS camera termed Temporal Ghost Fourier Compressive Inference Camera (TGIC), which provides a framework to minimize the data and computational burden simultaneously. The core of TGIC is co-design CVS encoding and machine vision algorithms both in optical domain. TGIC acquires pixel-wise temporal Fourier spectrum in one frame, and applies simple inverse fast Fourier transform algorithm to get the desired video. By implementing pre-designed optical Fourier sampling schemes, specific machine vision tasks can be accomplished in optical domain. In fact, the data captured by TGIC is the results of traditional machine vision algorithms derived from the video, therefore the computation resources will be greatly saved. In the experiments, we can recover 160 frames in 0.1s single exposure with 16x frame rate gain (periodic motion up to 2002 frames, 1000x frame rate gain), and typical machine vision applications such as moving object detection, tracking and extraction are also demonstrated.

[1]  Miles J. Padgett,et al.  Principles and prospects for single-pixel imaging , 2018, Nature Photonics.

[2]  Wesley E. Snyder,et al.  Demosaicking methods for Bayer color arrays , 2002, J. Electronic Imaging.

[3]  Jingang Zhong,et al.  Single-pixel imaging by means of Fourier spectrum acquisition , 2015, Nature Communications.

[4]  Frédo Durand,et al.  Visual vibrometry: Estimating material properties from small motions in video , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Guido Schuster,et al.  High spatio-temporal resolution video with compressed sensing. , 2015, Optics express.

[6]  Peter Kohl,et al.  Temporal Pixel Multiplexing for simultaneous high-speed high-resolution imaging , 2010, Nature Methods.

[7]  Zibang Zhang,et al.  Fast Fourier single-pixel imaging via binary illumination , 2017, Scientific Reports.

[8]  Wen Gao,et al.  Modeling Background and Segmenting Moving Objects from Compressed Video , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[9]  Sachin Sakhare,et al.  Image processing techniques for object tracking in video surveillance- A survey , 2015, 2015 International Conference on Pervasive Computing (ICPC).

[10]  Du-Ming Tsai,et al.  Motion detection using Fourier image reconstruction , 2008, Pattern Recognit. Lett..

[11]  Tae-Hyun Oh,et al.  Real-time motion detection based on Discrete Cosine Transform , 2012, 2012 19th IEEE International Conference on Image Processing.

[12]  Henrique S. Malvar,et al.  High-quality linear interpolation for demosaicing of Bayer-patterned color images , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[13]  Shree K. Nayar,et al.  Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Shree K. Nayar,et al.  Video from a single coded exposure photograph using a learned over-complete dictionary , 2011, 2011 International Conference on Computer Vision.

[15]  Guillermo Sapiro,et al.  Coded aperture compressive temporal imaging , 2013, Optics express.

[16]  Ari T. Friberg,et al.  Ghost imaging in the time domain , 2016, Nature Photonics.

[17]  Fabrice Devaux,et al.  Computational temporal ghost imaging , 2016, 1603.04647.

[18]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[19]  Ashok Veeraraghavan,et al.  Flexible Voxels for Motion-Aware Videography , 2010, ECCV.

[20]  Qionghai Dai,et al.  Efficient single pixel imaging in Fourier space , 2015, 1504.03823.

[21]  Qionghai Dai,et al.  Sinusoidal Sampling Enhanced Compressive Camera for High Speed Imaging , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Takeshi Takaki,et al.  Real-time laryngoscopic measurements of vocal-fold vibrations , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[23]  Frédo Durand,et al.  The visual microphone , 2014, ACM Trans. Graph..

[24]  Ramesh Raskar,et al.  Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Zibang Zhang,et al.  Hadamard single-pixel imaging versus Fourier single-pixel imaging. , 2017, Optics express.

[26]  Aswin C. Sankaranarayanan,et al.  Compressive Video Sensing: Algorithms, architectures, and applications , 2017, IEEE Signal Processing Magazine.