A High-Performance Hardware Architecture for a Frameless Stereo Vision Algorithm Implemented on a FPGA Platform

As a novelty, in this paper we present an event-based stereo vision matching approach based on time-correlation using segmentation to restrict the matching process to active image areas, exploiting the event-driven behavior of a silicon retina sensor. Stereo matching is used in depth generating camera systems for solving the correspondence problem and reconstructing 3D data. Using conventionally frame-based cameras, this correspondence problem is a time consuming and computationally expensive task. To overcome this issue, embedded systems can be used to speed up the calculation of stereo matching results. The silicon retina delivers asynchronous events if the illumination changes instead of synchronous intensity or color images. It provides sparse input data and therefore the output of the stereo vision algorithm (depth map) is also sparse. The high temporal resolution of such event-driven sensors leads to high data rates. To handle these and the correspondence problem in real time, we implemented our stereo matching algorithm for a field programmable gate array (FPGA). The results show that our matching criterion, based on the time of occurrence of an event, leads to a small average distance error and the parallel hardware architecture and efficient memory utilization results in a frame rate of up to 1140fps.

[1]  Kwabena Boahen,et al.  Retinomorphic chips that see quadruple images , 1999, Proceedings of the Seventh International Conference on Microelectronics for Neural, Fuzzy and Bio-Inspired Systems.

[2]  Tetsuya Yagi,et al.  Binocular robot vision emulating disparity computation in the primary visual cortex , 2008, Neural Networks.

[3]  Christoph Sulzbachner,et al.  Bio-inspired Stereo Vision System with Silicon Retina Imagers , 2009, ICVS.

[4]  Margrit Gelautz,et al.  Ground Truth Evaluation for Event-Based Silicon Retina Stereo Data , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[5]  C. Mead,et al.  Neuromorphic Robot Vision with Mixed Analog- Digital Architecture , 2005 .

[6]  Misha Mahowald,et al.  A silicon model of early visual processing , 1993, Neural Networks.

[7]  Kwabena Boahen,et al.  Optic nerve signals in a neuromorphic chip I: Outer and inner retina models , 2004, IEEE Transactions on Biomedical Engineering.

[8]  Tobi Delbrück,et al.  A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change , 2006, 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers.

[9]  Christoph Sulzbachner,et al.  Stereo-Vision Algorithm Based on Bio-Inspired Silicon Retinas for Implementation in Hardware , 2011, EUROCAST.

[10]  Christoph Sulzbachner,et al.  Event-Based Stereo Matching Approaches for Frameless Address Event Stereo Data , 2011, ISVC.

[11]  Kwabena Boahen,et al.  Optic nerve signals in a neuromorphic chip II: testing and results , 2004, IEEE Transactions on Biomedical Engineering.

[12]  Tobi Delbrück,et al.  Cooperative Stereo Matching Using Static and Dynamic Image Features , 1989, Analog VLSI Implementation of Neural Systems.

[13]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[14]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Misha Anne Mahowald,et al.  VLSI analogs of neuronal visual processing: a synthesis of form and function , 1992 .

[16]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[17]  Massimo A. Sivilotti,et al.  Wiring considerations in analog VLSI systems, with application to field-programmable networks , 1992 .

[18]  Daniel Matolin,et al.  A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS , 2011, IEEE Journal of Solid-State Circuits.

[19]  Ryad Benosman,et al.  Asynchronous Event-Based Hebbian Epipolar Geometry , 2011, IEEE Transactions on Neural Networks.

[20]  Ryad Benosman,et al.  Event-based 3D reconstruction from neuromorphic retinas , 2013, Neural Networks.

[21]  E. Culurciello,et al.  A biomorphic digital image sensor , 2003, IEEE J. Solid State Circuits.

[22]  Kwabena Boahen,et al.  Point-to-point connectivity between neuromorphic chips using address events , 2000 .

[23]  Ahmed Nabil Belbachir,et al.  Asynchronous Stereo Vision for Event-Driven Dynamic Stereo Sensor Using an Adaptive Cooperative Approach , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[24]  Stephan Schraml,et al.  Smartcam for real-time stereo vision - address-event based embedded system , 2007, VISAPP.

[25]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Daniel Matolin,et al.  An asynchronous time-based image sensor , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[27]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[28]  J. Harris,et al.  A Time-to-First-Spike CMOS Image Sensor , 2007 .

[29]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[30]  Christoph Sulzbachner,et al.  An optimized Silicon Retina stereo matching algorithm using time-space correlation , 2011, CVPR 2011 WORKSHOPS.

[31]  Bernabé Linares-Barranco,et al.  A Spatial Contrast Retina With On-Chip Calibration for Neuromorphic Spike-Based AER Vision Systems , 2007, IEEE Transactions on Circuits and Systems I: Regular Papers.

[32]  Tobi Delbrück,et al.  Asynchronous Event-Based Binocular Stereo Matching , 2012, IEEE Transactions on Neural Networks and Learning Systems.