Using FPGAs to perform embedded image registration

In computer vision, image registration is a technique used to find the relationship between two images of the same planar region in the world taken from different viewpoints due to the motion of a camera over time. Without a relationship of the coordinate space of one image to another, temporal information will not be fully exploitable, which generally reduces the applicable techniques to those that require only a single image to perform its function. In problems such as background subtraction, spatiotemporal information is commonly used [6, 22] to estimate the probability density function of each pixel in order to identify regions of interest in the scene, and without a pixel-to-pixel relationship between warped images this method cannot be applied. Tracking of objects found through detection is also complicated by camera motion as the motion of the object and camera are combined. At times, the spatial information provided by one image is not enough to perform the desired task, this can be remedied through the use of mosaicing, the merging of different images after they have been aligned, which allows for a moving camera to produce an increased field of view of a planar region. A similar problem is to increase the spatial resolution of an image through methods of super-resolution, which commonly use multiple images and image registration[13] to relate them to create an image of finer detail than those provided. Currently, image registration is being applied to many diverse problems: medical imaging[3], robotics, and image stabilization[11]. It is common in robotics that the results of the registration are used in autonomous control; therefore, the results must be produceable at the same rate that they are input (i.e., real-time). As image registration is generally not real-time, it is common to make assumptions (e.g., camera motion is only translation), approximations (e.g., use registration from down-sampled images), simplifications (e.g., reduce the input image size and frame-rate), and hardware architecture optimizations (e.g., DSPs). The goal of this research project is to develop an accurate real-time embedded image registration system that can be used in various problem domains (e.g., wearable computing, robotics, and surveillance) while investigating the natural tradeoffs between throughput, cost, generality, and accuracy. To design such a system, an image registration algorithm and complementing hardware architecture will be developed that can perform the algorithm in real-time and produce accurate results across problem domains and deliver the results in multiple formats to facilitate integration with other embedded image processing systems.

[1]  Liam Kilmartin,et al.  Xilinx FPGA implementation of a pixel processor for object detection applications , 2000 .

[2]  Mubarak Shah,et al.  Landing a UAV on a runway using image registration , 2008, 2008 IEEE International Conference on Robotics and Automation.

[3]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[4]  W. James MacLean,et al.  A Proposed Pipelined-Architecture for FPGA-Based Affine-Invariant Feature Detectors , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[5]  Jason Schlessman,et al.  Heterogeneous MPSoC Architectures for Embedded Computer Vision , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[6]  Michal Irani,et al.  Improving resolution by image registration , 1991, CVGIP Graph. Model. Image Process..

[7]  Bruce A. Draper,et al.  Accelerated image processing on FPGAs , 2003, IEEE Trans. Image Process..

[8]  Steve Mann,et al.  Video orbits of the projective group a simple approach to featureless estimation of parameters , 1997, IEEE Trans. Image Process..

[9]  Frederick M. Waltz,et al.  Fast software implementations of common 3x3 operators , 2005, SPIE Optics East.

[10]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[11]  Shoab Ahmad Khan,et al.  An optimized hardware accelerator for real time registration of aerial video imagery and its applications , 2004, Third International Conference on Image and Graphics (ICIG'04).

[12]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[13]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[14]  Cordelia Schmid,et al.  A Performance Evaluation of Local Descriptors , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  H. Opower.  Multiple view geometry in computer vision , 2002 .

[16]  Ramakant Nevatia,et al.  Efficient scan-window based object detection using GPGPU , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[17]  Michael Gleicher,et al.  Projective registration with difference decomposition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[19]  Jiri Matas,et al.  Robust wide-baseline stereo from maximally stable extremal regions , 2004, Image Vis. Comput..

[20]  Larry S. Davis,et al.  Non-parametric Model for Background Subtraction , 2000, ECCV.

[21]  Mamoru Nakanishi,et al.  A robust hardware algorithm for real-time object tracking in video sequences , 2004, Real Time Imaging.

[22]  Mubarak Shah,et al.  COCOA: tracking in aerial imagery , 2006, SPIE Defense + Commercial Sensing.

[23]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[24]  Richard Szeliski,et al.  Construction of Panoramic Image Mosaics with Global and Local Alignment , 2001 .

[25]  C. T. Johnston.  Implementing Image Processing Algorithms on FPGAs , 2005 .

[26]  Kristin J. Dana,et al.  Real-time scene stabilization and mosaic construction , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[27]  Raimundo Carlos Silvério Freire,et al.  FPGA architecture for static background subtraction in real time , 2006, SBCCI '06.

[28]  Edward H. Adelson,et al.  Spatio-temporal segmentation of video data , 1994, Electronic Imaging.

[29]  S.A. Khan,et al.  Hardware design for registration of aerial video imagery , 2004, 8th International Multitopic Conference, 2004. Proceedings of INMIC 2004..

[30]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[31]  Vincent Lepetit,et al.  A fast local descriptor for dense matching , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Ronald F. DeMara,et al.  Autonomous FPGA fault handling through competitive runtime reconfiguration , 2005, 2005 NASA/DoD Conference on Evolvable Hardware (EH'05).

[33]  Andrew Hunter,et al.  A single-chip FPGA implementation of real-time adaptive background model , 2005, Proceedings. 2005 IEEE International Conference on Field-Programmable Technology, 2005..

[34]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[35]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).