Robust Real-Time Super-Resolution on FPGA and an Application to Video Enhancement

The high density image sensors of state-of-the-art imaging systems provide outputs with high spatial resolution, but require long exposure times. This limits their applicability, due to the motion blur effect. Recent technological advances have lead to adaptive image sensors that can combine several pixels together in real time to form a larger pixel. Larger pixels require shorter exposure times and produce high-frame-rate samples with reduced motion blur. This work proposes combining an FPGA with an adaptive image sensor to produce an output of high resolution both in space and time. The FPGA is responsible for the spatial resolution enhancement of the high-frame-rate samples using super-resolution (SR) techniques in real time. To achieve it, this article proposes utilizing the Iterative Back Projection (IBP) SR algorithm. The original IBP method is modified to account for the presence of noise, leading to an algorithm more robust to noise. An FPGA implementation of this algorithm is presented. The proposed architecture can serve as a general purpose real-time resolution enhancement system, and its performance is evaluated under various noise levels.

[1]  Jan Flusser,et al.  A Unified Approach to Superresolution and Multichannel Blind Deconvolution , 2007, IEEE Transactions on Image Processing.

[2]  J.-Y. Bouguet,et al.  Pyramidal implementation of the lucas kanade feature tracker , 1999 .

[3]  George A. Constantinides,et al.  A Floating-point Extended Kalman Filter Implementation for Autonomous Mobile Robots , 2009, 2007 International Conference on Field Programmable Logic and Applications.

[4]  A. El Gamal,et al.  CMOS image sensors , 2005, IEEE Circuits and Devices Magazine.

[5]  Feng Xiao,et al.  Resolution and light sensitivity tradeoff with pixel size , 2006, Electronic Imaging.

[6]  Michael Elad,et al.  Advances and challenges in super‐resolution , 2004, Int. J. Imaging Syst. Technol..

[7]  Yang Liu,et al.  Efficient Mapping of a Kalman Filter into an FPGA using Taylor Expansion , 2007, 2007 International Conference on Field Programmable Logic and Applications.

[8]  Greg Welch,et al.  An Introduction to Kalman Filter , 1995, SIGGRAPH 2001.

[9]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[10]  Moon Gi Kang,et al.  Super-resolution image reconstruction: a technical overview , 2003, IEEE Signal Process. Mag..

[11]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[12]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[13]  Brian A. Wandell,et al.  How small should pixel size be? , 2000, Electronic Imaging.

[14]  Shree K. Nayar,et al.  Motion-based motion deblurring , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  E. Fossum,et al.  CMOS active pixel image sensors for highly integrated imaging systems , 1997, IEEE J. Solid State Circuits.

[16]  J. L. Roux.  An Introduction to the Kalman Filter , 2003 .

[17]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[18]  Yaron Caspi,et al.  Under the supervision of , 2003 .

[19]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[20]  Takeo Kanade,et al.  Limits on super-resolution and how to break them , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[21]  N S Kopeika,et al.  Enhanced-resolution image restoration from a sequence of low-frequency vibrated images by use of convex projections. , 2001, Applied optics.

[22]  Timothy G. Constandinou,et al.  An adaptable foveating vision chip , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[23]  H Stark,et al.  High-resolution image recovery from image-plane arrays, using convex projections. , 1989, Journal of the Optical Society of America. A, Optics and image science.

[24]  Nitin Sampat,et al.  Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications , 2000 .

[25]  D. Youla,et al.  Image Restoration by the Method of Convex Projections: Part 1ߞTheory , 1982, IEEE Transactions on Medical Imaging.

[26]  Maria E. Angelopoulou,et al.  Video enhancement on an adaptive image sensor , 2008, 2008 15th IEEE International Conference on Image Processing.

[27]  Maria E. Angelopoulou,et al.  FPGA-based Real-time Super-Resolution on an Adaptive Image Sensor , 2008, ARC.

[28]  Rudolph van der Merwe,et al.  The unscented Kalman filter for nonlinear estimation , 2000, Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No.00EX373).

[29]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Abbas El Gamal,et al.  Analysis of temporal noise in CMOS photodiode active pixel sensor , 2001, IEEE J. Solid State Circuits.