Pipelined architecture for real-time cost-optimized extraction of visual primitives based on FPGAs

This paper presents an architecture for the extraction of visual primitives on chip: energy, orientation, disparity, and optical flow. This cost-optimized architecture processes in real time high-resolution images for real-life applications. In fact, we present a versatile architecture that may be customized for different performance requirements depending on the target application. In this case, dedicated hardware and its potential on-chip implementation on FPGA devices become an efficient solution. We have developed a multi-scale approach for the computation of the gradient-based primitives. Gradient-based methods are very popular in the literature because they provide a very competitive accuracy vs. efficiency trade-off. The hardware implementation of the system is performed using superscalar fine-grain pipelines to exploit the maximum degree of parallelism provided by the FPGA. The system reaches 350 and 270 VGA frames per second (fps) for the disparity and optical flow computations respectively in their mono-scale version and up to 32 fps for the multi-scale scheme extracting all the described features in parallel. In this work we also analyze the performance in accuracy and hardware resources of the proposed implementation.

[1]  Johan Wiklund,et al.  Multidimensional Orientation Estimation with Applications to Texture Analysis and Optical Flow , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Eduardo Ros,et al.  High-Performance Optical-Flow Architecture Based on a Multi-Scale, Multi-Orientation Phase-Based Model , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Theocharis Theocharides,et al.  Towards hardware stereoscopic 3D reconstruction a real-time FPGA computation of the disparity map , 2010, 2010 Design, Automation & Test in Europe Conference & Exhibition (DATE 2010).

[5]  Antonio Cañas,et al.  Hardware description of multi-layer perceptrons with different abstraction levels , 2006, Microprocess. Microsystems.

[6]  Javier Díaz,et al.  Fine grain pipeline architecture for high performance phase-based optical flow computation , 2010, J. Syst. Archit..

[7]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[8]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[9]  Abbas El Gamal,et al.  Optical flow estimation using high frame rate sequences , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[10]  Timo Kohlberger,et al.  A Multigrid Platform for Real-Time Motion Computation with Discontinuity-Preserving Variational Methods , 2006, International Journal of Computer Vision.

[11]  Marc M. Van Hulle,et al.  Realtime phase-based optical flow on the GPU , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[12]  Andreas Steininger,et al.  SAD-Based Stereo Matching Using FPGAs , 2009 .

[13]  Dah-Jye Lee,et al.  Real-Time Optical Flow Calculations on FPGA and GPU Architectures: A Comparison Study , 2008, 2008 16th International Symposium on Field-Programmable Custom Computing Machines.

[14]  Javier Díaz,et al.  Superpipelined high-performance optical-flow computation architecture , 2008, Comput. Vis. Image Underst..

[15]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[16]  Mancia Anguita,et al.  Optimization Strategies for High-Performance Computing of Optical-Flow in General-Purpose Processors , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[17]  Eduardo Ros Vidal,et al.  Robust Bioinspired Architecture for Optical-Flow Computation , 2010, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[18]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[19]  Hans-Hellmut Nagel,et al.  Optical Flow Estimation: Advances and Comparisons , 1994, ECCV.

[20]  Edward H. Adelson,et al.  The Laplacian Pyramid as a Compact Image Code , 1983, IEEE Trans. Commun..

[21]  Ines Ernst,et al.  Mutual Information Based Semi-Global Stereo Matching on the GPU , 2008, ISVC.

[22]  Ioannis Andreadis,et al.  A Real-Time Occlusion Aware Hardware Structure for Disparity Map Computation , 2009, ICIAP.

[23]  Dah-Jye Lee,et al.  A Fast and Accurate Tensor-based Optical Flow Algorithm Implemented in FPGA , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[24]  Jean-Guy Fontaine,et al.  High parallel disparity map computing on FPGA , 2010, Proceedings of 2010 IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications.

[25]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[26]  Jonathan W. Brandt,et al.  Improved Accuracy in Gradient-Based Optical Flow Estimation , 1997, International Journal of Computer Vision.

[27]  P. Anandan,et al.  Hierarchical Model-Based Motion Estimation , 1992, ECCV.

[28]  Paul Merrell,et al.  Structure from motion using optical flow probability distributions , 2005, SPIE Defense + Commercial Sensing.

[29]  Lei Chen,et al.  An FPGA-based RGBD imager , 2011, Machine Vision and Applications.

[30]  Peter I. Corke,et al.  Quantitative Evaluation of Matching Methods and Validity Measures for Stereo Vision , 2001, Int. J. Robotics Res..

[31]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[32]  Wei Zhang,et al.  Moving vehicles detection based on adaptive motion histogram , 2010, Digit. Signal Process..

[33]  Andrea Giachetti,et al.  The use of optical flow for the autonomous navigation , 1994, ECCV.

[34]  Vittorio Murino,et al.  A real-time versatile roadway path extraction and tracking on an FPGA platform , 2010, Comput. Vis. Image Underst..

[35]  Eduardo Ros,et al.  Massive Parallel-Hardware Architecture for Multiscale Stereo, Optical Flow and Image-Structure Computation , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[36]  Javier Díaz,et al.  Multi-port abstraction layer for FPGA intensive memory exploitation applications , 2010, J. Syst. Archit..

[37]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[38]  Matthew A. Brown,et al.  Unsupervised 3D object recognition and reconstruction in unordered datasets , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[39]  Larry Matthies,et al.  FPGA implementation of stereo disparity with high throughput for mobility applications , 2011, 2011 Aerospace Conference.

[40]  Lei Chen,et al.  A Parallel Reconfigurable Architecture for Real-Time Stereo Vision , 2009, 2009 International Conference on Embedded Software and Systems.

[41]  Eduardo Ros,et al.  A Comparison of FPGA and GPU for Real-Time Phase-Based Optical Flow, Stereo, and Local Image Features , 2012, IEEE Transactions on Computers.

[42]  Yasuo Kuniyoshi,et al.  Velocity and disparity cues for robust real-time binocular tracking , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[43]  Joachim Weickert,et al.  Variational optic flow on the Sony PlayStation 3 , 2010, Journal of Real-Time Image Processing.

[44]  Adel M. Alimi,et al.  Event Detection from Video Surveillance Data Based on Optical Flow Histogram and High-level Feature Extraction , 2009, 2009 20th International Workshop on Database and Expert Systems Application.

[45]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[46]  Eero P. Simoncelli.  Design of multi-dimensional derivative filters , 1994, Proceedings of 1st International Conference on Image Processing.

[47]  Rama Chellappa,et al.  Accuracy vs. Efficiency Trade-offs in Optical Flow Algorithms , 1996, ECCV.

[48]  Alan Yuille,et al.  Active Vision , 2014, Computer Vision, A Reference Guide.

[49]  Brendan McCane,et al.  On Benchmarking Optical Flow , 2001, Comput. Vis. Image Underst..

[50]  Jae Wook Jeon,et al.  Pipelined Hardware Architecture for High-Speed Optical Flow Estimation Using FPGA , 2010, 2010 18th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines.

[51]  Andrej Zemva,et al.  FPGA-oriented HW/SW implementation of ECG beat detection and classification algorithm , 2010, Digit. Signal Process..

[52]  Javier Díaz,et al.  FPGA-based real-time optical-flow system , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[53]  Timo Kohlberger,et al.  Domain decomposition for variational optical-flow computation , 2005, IEEE Transactions on Image Processing.