Efficient Hardware Implementation of the Horn-Schunck Algorithm for High-Resolution Real-Time Dense Optical Flow Sensor

This article presents an efficient hardware implementation of the Horn-Schunck algorithm that can be used in an embedded optical flow sensor. An architecture is proposed, that realises the iterative Horn-Schunck algorithm in a pipelined manner. This modification allows to achieve data throughput of 175 MPixels/s and makes processing of Full HD video stream (1, 920 × 1, 080 @ 60 fps) possible. The structure of the optical flow module as well as pre- and post-filtering blocks and a flow reliability computation unit is described in details. Three versions of optical flow modules, with different numerical precision, working frequency and obtained results accuracy are proposed. The errors caused by switching from floating- to fixed-point computations are also evaluated. The described architecture was tested on popular sequences from an optical flow dataset of the Middlebury University. It achieves state-of-the-art results among hardware implementations of single scale methods. The designed fixed-point architecture achieves performance of 418 GOPS with power efficiency of 34 GOPS/W. The proposed floating-point module achieves 103 GFLOPS, with power efficiency of 24 GFLOPS/W. Moreover, a 100 times speedup compared to a modern CPU with SIMD support is reported. A complete, working vision system realized on Xilinx VC707 evaluation board is also presented. It is able to compute optical flow for Full HD video stream received from an HDMI camera in real-time. The obtained results prove that FPGA devices are an ideal platform for embedded vision systems.

[1]  Wen Gao,et al.  Learning discriminative features for fast frame-based action recognition , 2013, Pattern Recognit..

[2]  Unai Bidarte,et al.  Hardware implementation of optical flow constraint equation using FPGAs , 2005, Comput. Vis. Image Underst..

[3]  Zbigniew Mikrut,et al.  Combining Pattern Matching and Optical Flow Methods in Home Care Vision System , 2012, ITIB.

[4]  Nahum Kiryati,et al.  Coarse to over-fine optical flow estimation , 2007, Pattern Recognit..

[5]  Rubén Hernández,et al.  Real-time Optical-flow Computation for Motion Estimation under Varying Illumination Conditions , 2012 .

[6]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[7]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[9]  Eduardo Ros,et al.  High-Performance Optical-Flow Architecture Based on a Multi-Scale, Multi-Orientation Phase-Based Model , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Henrik I. Christensen,et al.  Constrained structure and motion estimation from optical flow , 2002, Object recognition supported by user interaction for service robots.

[11]  E. M. Saad,et al.  A reliable FPGA-based real-time optical-flow estimation , 2009, 2009 National Radio Science Conference.

[12]  Björn Kruse,et al.  A Parallel Picture Processing Machine , 1973, IEEE Transactions on Computers.

[13]  Gert F. Trommer,et al.  Development of a method for image-based motion estimation of a VTOL-MAV on FPGA , 2011, Proceedings of the 2011 Conference on Design & Architectures for Signal & Image Processing (DASIP).

[14]  H. S. Wolff,et al.  iRun: Horizontal and Vertical Shape of a Region-Based Graph Compression , 2022, Sensors.

[15]  Robin R. Murphy,et al.  A VLSI Architecture and Algorithm for Lucas–Kanade-Based Optical Flow Computation , 2010, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[16]  N. H. Hamid,et al.  FPGA-based hardware implementation of optical flow constraint equation of Horn and Schunck , 2012, 2012 4th International Conference on Intelligent and Advanced Systems (ICIAS2012).

[17]  Eduardo Ros Vidal,et al.  Robust Bioinspired Architecture for Optical-Flow Computation , 2010, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[18]  Md Atiqur Rahman Ahad,et al.  Action recognition algorithm based on optical flow and RANSAC in frequency domain , 2011, SICE Annual Conference 2011.

[19]  Unai Bidarte,et al.  Optical Flow Estimator Using VHDL for Implementation in FPGA , .

[20]  J. Weickert,et al.  A Confidence Measure for Variational Optic flow Methods , 2006 .

[21]  José Luis Martín,et al.  Hardware architecture for optical flow estimation in real time , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[22]  Angel Domingo Sappa,et al.  An empirical study on optical flow accuracy depending on vehicle speed , 2012, 2012 IEEE Intelligent Vehicles Symposium.

[23]  B. McCane,et al.  On the Evaluation of Optical Flow Algorithms , 1998 .

[24]  Jose Antonio Boluda,et al.  Change-driven data flow image processing architecture for optical flow computation , 2007, Journal of Real-Time Image Processing.

[25]  M. R. B. Bahar,et al.  High performance implementation of the Horn and Schunck optical flow algorithm on FPGA , 2012, 20th Iranian Conference on Electrical Engineering (ICEE2012).

[26]  Y. Zinbi,et al.  Moving object Segmentation; using optical flow with active contour model , 2008, 2008 3rd International Conference on Information and Communication Technologies: From Theory to Applications.

[27]  Afsar Saranli,et al.  An FPGA based high performance optical flow hardware design for computer vision applications , 2013, Microprocess. Microsystems.

[28]  V. Pisarevsky,et al.  Intel's Computer Vision Library: applications in calibration, stereo segmentation, tracking, gesture, face and object recognition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[29]  Marc M. Van Hulle,et al.  Realtime phase-based optical flow on the GPU , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[30]  Javier Díaz,et al.  Superpipelined high-performance optical-flow computation architecture , 2008, Comput. Vis. Image Underst..

[31]  Zbigniew Mikrut,et al.  Video Detection Algorithm Using an Optical Flow Calculation Method , 2012, MCSS.

[32]  M. V. Wilkes,et al.  The Art of Computer Programming, Volume 3, Sorting and Searching , 1974 .

[33]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[34]  Hafiz Imtiaz,et al.  An optical flow based approach for action recognition , 2011, 14th International Conference on Computer and Information Technology (ICCIT 2011).

[35]  Donald G. Bailey,et al.  Design for Embedded Image Processing on FPGAs , 2011 .

[36]  Eduardo Ros,et al.  Massive Parallel-Hardware Architecture for Multiscale Stereo, Optical Flow and Image-Structure Computation , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[37]  Javier Díaz,et al.  Parallel Architecture for Hierarchical Optical Flow Estimation Based on FPGA , 2012, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[38]  A. Paulo Moreira,et al.  Revisiting Lucas-Kanade and Horn-Schunck , 2013 .

[39]  Yassine Ruichek,et al.  Object tracking using Harris corner points based optical flow propagation and Kalman filter , 2011, 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[40]  Coskun Bayrak,et al.  Sports video summarization based on motion analysis , 2013, Comput. Electr. Eng..

[41]  M. Gorgon.  Parallel performance of the fine-grain pipeline FPGA image processing system , 2012 .

[42]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[43]  Timothy J. Purcell.  Sorting and searching , 2005, SIGGRAPH Courses.

[44]  Mancia Anguita,et al.  Optimization Strategies for High-Performance Computing of Optical-Flow in General-Purpose Processors , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[45]  Tomasz Kryjak,et al.  Real-time background generation and foreground object segmentation for high-definition colour video stream in FPGA device , 2012, Journal of Real-Time Image Processing.

[46]  Zhan Ma,et al.  Smart Cameras , 2020, Encyclopedia of Biometrics.