Dynamic Non-Regular Sampling Sensor Using Frequency Selective Reconstruction

Both a high spatial and a high temporal resolution of images and videos are desirable in many applications, such as entertainment systems, monitoring manufacturing processes, or video surveillance. Due to the limited throughput of pixels per second, however, there is always a tradeoff between acquiring sequences with a high spatial resolution at a low temporal resolution or vice versa. In this paper, a modified sensor concept is proposed which is able to acquire both a high spatial and a high temporal resolution. This is achieved by dynamically reading out only a subset of pixels in a non-regular order to obtain a high temporal resolution. A full high spatial resolution is then obtained by performing a subsequent 3D reconstruction of the partially acquired frames. The main benefit of the proposed dynamic readout is that for each frame, different sampling points are available, which is advantageous since this information can significantly enhance the reconstruction quality of the proposed reconstruction algorithm. Using the proposed dynamic readout strategy, gains in the peak-signal-to-noise ratio (PSNR) of up to 8.55 dB are achieved compared with a static readout strategy. Compared with the other state-of-the-art techniques, such as frame rate up-conversion or super-resolution, which are also able to reconstruct sequences with both a high spatial and a high temporal resolution, average gains in PSNR of up to 6.58 dB are possible.

[1]  Andreas K. Maier,et al.  Robust Multiframe Super-Resolution Employing Iteratively Re-Weighted Minimization , 2016, IEEE Transactions on Computational Imaging.

[2]  André Kaup,et al.  Spatiotemporal Selective Extrapolation for 3-D Signals and Its Applications in Video Communications , 2007, IEEE Transactions on Image Processing.

[3]  André Kaup,et al.  Fast orthogonality deficiency compensation for improved frequency selective image extrapolation , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[4]  Peter Kohl,et al.  Temporal Pixel Multiplexing for simultaneous high-speed high-resolution imaging , 2010, Nature Methods.

[5]  André Kaup,et al.  Increasing imaging resolution by covering your sensor , 2011, 2011 18th IEEE International Conference on Image Processing.

[6]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[7]  Peyman Milanfar,et al.  Kernel Regression for Image Processing and Reconstruction , 2007, IEEE Transactions on Image Processing.

[8]  Michael Elad,et al.  Fast and robust multiframe super resolution , 2004, IEEE Transactions on Image Processing.

[9]  André Kaup,et al.  Reducing randomness of non-regular sampling masks for image reconstruction , 2014, 2014 IEEE Visual Communications and Image Processing Conference.

[10]  André Kaup,et al.  Content-Adaptive Motion Compensated Frequency Selective Extrapolation for error concealment in video communication , 2010, 2010 IEEE International Conference on Image Processing.

[11]  Jordi Salvador,et al.  Naive Bayes Super-Resolution Forest , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  André Kaup,et al.  Optimized processing order for 3D hole filling in video sequences using frequency selective extrapolation , 2016, 2016 Picture Coding Symposium (PCS).

[13]  D. F. Watson,et al.  Contouring: a guide to the analysis and display of spatial data (with programs on diskette). , 1992 .

[14]  André Kaup,et al.  Motion compensated frame rate up-conversion using 3D frequency selective extrapolation and a multi-layer consistency check , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[15]  Adel M. Alimi,et al.  Resolution enhancement of textual images: a survey of single image-based methods , 2016, IET Image Process..

[16]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[17]  André Kaup,et al.  Complex-Valued Frequency Selective Extrapolation for Fast Image and Video Signal Extrapolation , 2010, IEEE Signal Processing Letters.

[18]  André Kaup,et al.  Reconstruction of images taken by a pair of non-regular sampling sensors using correlation based matching , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[19]  Thomas B. Moeslund,et al.  Super-resolution: a comprehensive survey , 2014, Machine Vision and Applications.

[20]  Felix J. Herrmann,et al.  Irregular sampling: from aliasing to noise , 2007 .

[21]  André Kaup,et al.  Resampling Images to a Regular Grid From a Non-Regular Subset of Pixel Positions Using Frequency Selective Reconstruction , 2015, IEEE Transactions on Image Processing.

[22]  Kaveh Kangarloo,et al.  A Survey on Super-Resolution Methods for Image Reconstruction , 2014 .

[23]  Damien Garcia,et al.  Robust smoothing of gridded data in one and higher dimensions with missing values , 2010, Comput. Stat. Data Anal..

[24]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Michael B. Wakin.  Sparse Image and Signal Processing: Wavelets, Curvelets, Morphological Diversity (Starck, J.-L., et al; 2010) [Book Reviews] , 2011, IEEE Signal Processing Magazine.

[26]  André Kaup,et al.  Reconstruction of videos taken by a non-regular sampling sensor , 2015, 2015 Visual Communications and Image Processing (VCIP).

[27]  André Kaup,et al.  Texture-dependent frequency selective reconstruction of non-regularly sampled images , 2022, 2016 Picture Coding Symposium (PCS).

[28]  M. Mori,et al.  1/4-inch 2-mpixel MOS image sensor with 1.75 transistors/pixel , 2004, IEEE Journal of Solid-State Circuits.

[29]  J. Akita,et al.  A CMOS image sensor with pseudorandom pixel placement for clear imaging , 2009, 2009 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS).

[30]  Lei Zhang,et al.  Nonlocally Centralized Sparse Representation for Image Restoration , 2013, IEEE Transactions on Image Processing.

[31]  André Kaup,et al.  Sparsity-based defect pixel compensation for arbitrary camera raw images , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[32]  Soheil Darabi,et al.  A novel framework for imaging using compressed sensing , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[33]  Michael Schöberl,et al.  Diffraction and photometric limits in today's miniature digital camera systems , 2013, Photonics West - Micro and Nano Fabricated Electromechanical and Optical Components.