Bio-Inspired Stereo Vision Calibration for Dynamic Vision Sensors

Many advances have been made in the field of computer vision. Several recent research trends have focused on mimicking human vision by using a stereo vision system. In multi-camera systems, a calibration process is usually implemented to improve the results accuracy. However, these systems generate a large amount of data to be processed; therefore, a powerful computer is required and, in many cases, this cannot be done in real time. Neuromorphic Engineering attempts to create bio-inspired systems that mimic the information processing that takes place in the human brain. This information is encoded using pulses (or spikes) and the generated systems are much simpler (in computational operations and resources), which allows them to perform similar tasks with much lower power consumption, thus these processes can be developed over specialized hardware with real-time processing. In this work, a bio-inspired stereo-vision system is presented, where a calibration mechanism for this system is implemented and evaluated using several tests. The result is a novel calibration technique for a neuromorphic stereo vision system, implemented over specialized hardware (FPGA - Field-Programmable Gate Array), which allows obtaining reduced latencies on hardware implementation for stand-alone systems, and working in real time.

[1]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[2]  Joseph L. Mundy,et al.  Relationship between photogrammmetry and computer vision , 1993, Defense, Security, and Sensing.

[3]  Ryad Benosman,et al.  Event-Driven Stereo Visual Tracking Algorithm to Solve Object Occlusion , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Rüdiger Dillmann,et al.  Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms , 2019, Front. Neurorobot..

[5]  Ke Zhu,et al.  Computational optimized 3D reconstruction system for airborne image sequences , 2010 .

[6]  Sandro Barone,et al.  3D Reconstruction and Restoration Monitoring of Sculptural Artworks by a Multi-Sensor Framework , 2012, Sensors.

[7]  Alejandro Linares-Barranco,et al.  Stereo Matching: From the Basis to Neuromorphic Engineering , 2012 .

[8]  M. Domínguez-Morales,et al.  Image matching algorithms in stereo vision using address-event-representation: A theoretical study and evaluation of the different algorithms , 2011, Proceedings of the International Conference on Signal Processing and Multimedia Applications.

[9]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[10]  Bernabé Linares-Barranco,et al.  Multicasting Mesh AER: A Scalable Assembly Approach for Reconfigurable Neuromorphic Structured AER Systems. Application to ConvNets , 2013, IEEE Transactions on Biomedical Circuits and Systems.

[11]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  M. Domínguez-Morales,et al.  Live demonstration: On the distance estimation of moving targets with a Stereo-Vision AER system , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[13]  John N. Lygouras,et al.  Robotic Mapping and Localization with Real-Time Dense Stereo on Reconfigurable Hardware , 2010, Int. J. Reconfigurable Comput..

[14]  L D Harmon,et al.  Picture processing by computer. , 1969, Science.

[15]  Tobi Delbrück,et al.  A 5 Meps $100 USB2.0 Address-Event Monitor-Sequencer Interface , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[16]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[17]  Fernando Díaz del Río,et al.  Neuromorphic Real-Time Objects Tracking Using Address Event Representation and Silicon Retina , 2011, IWANN.

[18]  Helder Araújo,et al.  Iterative multistep explicit camera calibration , 1999, IEEE Trans. Robotics Autom..

[19]  Paul R. Cohen,et al.  Camera Calibration with Distortion Models and Accuracy Evaluation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Ivan Detchev,et al.  Practical In Situ Implementation of a Multicamera Multisystem Calibration , 2018, J. Sensors.

[21]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[22]  Massimo A. Sivilotti,et al.  Wiring considerations in analog VLSI systems, with application to field-programmable networks , 1992 .

[23]  Tobi Delbrück,et al.  A USB3.0 FPGA event-based filtering and tracking framework for dynamic vision sensors , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[24]  Juan López Coronado,et al.  Neuro-Inspired Spike-Based Motion: From Dynamic Vision Sensor to Robot Motor Open-Loop Control through Spike-VITE , 2013, Sensors.

[25]  Alexander Andreopoulos,et al.  A Low Power, High Throughput, Fully Event-Based Stereo System , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Bernabé Linares-Barranco,et al.  A Configurable Event-Driven Convolutional Node with Rate Saturation Mechanism for Modular ConvNet Systems Implementation , 2018, Front. Neurosci..

[27]  M. Domínguez-Morales,et al.  On the Designing of Spikes Band-Pass Filters for FPGA , 2011, ICANN.

[28]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[29]  James Black,et al.  Multi view image surveillance and tracking , 2002, Workshop on Motion and Video Computing, 2002. Proceedings..

[30]  M. Domínguez-Morales,et al.  An Approach to Distance Estimation with Stereo Vision Using Address-Event-Representation , 2011, ICONIP.

[31]  Francisco Angel Moreno,et al.  Stereo vision specific models for particle filter-based SLAM , 2009, Robotics Auton. Syst..

[32]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[33]  Soon-Yong Park,et al.  An Efficient Calibration Method for a Stereo Camera System with Heterogeneous Lenses Using an Embedded Checkerboard Pattern , 2017, J. Sensors.

[34]  Manuel Mazo,et al.  Stereo Vision Tracking of Multiple Objects in Complex Indoor Environments , 2010, Sensors.

[35]  G. Jimenez-Moreno,et al.  NAVIS: Neuromorphic Auditory VISualizer Tool , 2017, Neurocomputing.

[36]  Steve B. Furber,et al.  Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype , 2018, Front. Neurosci..

[37]  Robert B. Kelley,et al.  Camera Models Based on Data from Two Calibration Planes , 1981 .

[38]  Christoph Sulzbachner,et al.  Address-Event Based Stereo Vision with Bio-Inspired Silicon Retina Imagers , 2011 .

[39]  C. Dyer.  Volumetric Scene Reconstruction from Multiple Views , 2001 .

[40]  Bernabé Linares-Barranco,et al.  On Multiple AER Handshaking Channels Over High-Speed Bit-Serial Bidirectional LVDS Links With Flow-Control and Clock-Correction on Commercial FPGAs for Scalable Neuromorphic Systems , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[41]  M. Domínguez-Morales,et al.  Simulating Building Blocks for Spikes Signals Processing , 2011, IWANN.

[42]  M. Domínguez-Morales,et al.  A Neuro-Inspired Spike-Based PID Motor Controller for Multi-Motor Robots with Low Cost FPGAs , 2012, Sensors.

[43]  Tobi Delbrück,et al.  Asynchronous Event-Based Binocular Stereo Matching , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[44]  Juan Pedro Dominguez-Morales,et al.  Stereo Matching in Address-Event-Representation (AER) Bio-Inspired Binocular Systems in a Field-Programmable Gate Array (FPGA) , 2019 .

[45]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[46]  Yuxia Yang,et al.  Study on semi-global matching algorithm extended for multi baseline matching and parallel processing method based on GPU , 2014 .

[47]  M. Domínguez-Morales,et al.  On the AER Stereo-Vision Processing: A Spike Approach to Epipolar Matching , 2013, ICONIP.

[48]  Aly A. Farag,et al.  Neurocalibration: a neural network that can tell camera calibration parameters , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[49]  Tobi Delbrück,et al.  Approaching Retinal Ganglion Cell Modeling and FPGA Implementation for Robotics , 2018, Entropy.

[50]  Fernando Díaz del Río,et al.  Real time multiple objects tracking based on a bio-inspired processing cascade architecture , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[51]  S. P. Mudur,et al.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[52]  Tobi Delbrück,et al.  CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory–Processing– Learning–Actuating System for High-Speed Visual Object Recognition and Tracking , 2009, IEEE Transactions on Neural Networks.

[53]  Ricardo Tapiador-Morales,et al.  Neuromorphic LIF Row-by-Row Multiconvolution Processor for FPGA , 2019, IEEE Transactions on Biomedical Circuits and Systems.

[54]  Janne Heikkilä,et al.  Geometric Camera Calibration Using Circular Control Points , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Jörg Conradt,et al.  Spiking Cooperative Stereo-Matching at 2 ms Latency with Neuromorphic Hardware , 2017, Living Machines.

[56]  Tobi Delbrück,et al.  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output , 2014, Proceedings of the IEEE.

[57]  O. Faugeras.  Three-dimensional computer vision: a geometric viewpoint , 1993 .

[58]  Elena Cerezuela-Escudero,et al.  A Binaural Neuromorphic Auditory Sensor for FPGA: A Spike Signal Processing Approach , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[59]  Ryad Benosman,et al.  Event-based 3D reconstruction from neuromorphic retinas , 2013, Neural Networks.