Asynchronous Event-Based 3 D Reconstruction From Neuromorphic Retinas

This paper presents a novel N-ocular 3D reconstruction algorithm for eventbased vision data from bio-inspired artificial retina sensors. Artificial restinas capture visual information asynchronously and encode it into streams of asynchronous spike-like pulse signals carrying information on e.g. temporal contrast events in the scene. The precise time of the occurence of these visual features are implicitly encoded in the spike timings. Due to the high temporal resolution of the asynchronous visual information acquisition, the output of these sensors is ideally suited for dynamic 3D reconstruction. The presented technique takes full benefit of the event-driven operation, i.e. events are processed individually at the moment they arrive. This strategy allows to preserve the original dynamics of the scene, hence allowing for more robust 3D reconstructions. As opposed to existing techniques, this algorithm is based on geometric and time constraints alone, making it particularly simple to implement and largely linear.

[1]  Misha Anne Mahowald,et al.  VLSI analogs of neuronal visual processing: a synthesis of form and function , 1992 .

[2]  R Perez,et al.  Neural mechanisms underlying stereoscopic vision , 1998, Progress in Neurobiology.

[3]  Pavel Krsek,et al.  Robust Euclidean alignment of 3D point sets: the trimmed iterative closest point algorithm , 2005, Image Vis. Comput..

[4]  Zhi-Gang Zheng,et al.  A region based stereo matching algorithm using cooperative optimization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Steven M. Seitz,et al.  Shape and materials by example: a photometric stereo approach , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Tetsuya Yagi,et al.  Binocular robot vision emulating disparity computation in the primary visual cortex , 2008, Neural Networks.

[7]  Marc Levoy,et al.  Better optical triangulation through spacetime analysis , 1995, Proceedings of IEEE International Conference on Computer Vision.

[8]  W Skrandies,et al.  An early antecedent to modern random dot stereograms --'the secret stereoscopic writing' of Ramón y Cajal. , 2000, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[9]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[10]  Miao Liao,et al.  High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[11]  Steve Marschner,et al.  Dual photography , 2005, ACM Trans. Graph..

[12]  Shree K. Nayar,et al.  Projection defocus analysis for scene capture and image display , 2006, SIGGRAPH 2006.

[13]  Li Xu,et al.  Stereo Matching: An Outlier Confidence Approach , 2008, ECCV.

[14]  Ryad Benosman,et al.  Asynchronous Event-Based Hebbian Epipolar Geometry , 2011, IEEE Transactions on Neural Networks.

[15]  Tobi Delbrück,et al.  Live demonstration: Gesture-based remote control using stereo pair of dynamic vision sensors , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[16]  Parvati Dev,et al.  Perception of Depth Surfaces in Random-Dot Stereograms: A Neural Model , 1975, Int. J. Man Mach. Stud..

[17]  Carver A. Mead,et al.  Analog VLSI Phototransduction by continuous-time, adaptive, logarithmic photoreceptor circuits , 1995 .

[18]  Olga Veksler,et al.  Stereo correspondence by dynamic programming on a tree , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[19]  Szymon Rusinkiewicz,et al.  Viewpoint-Coded Structured Light , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Li Zhang,et al.  Rapid shape acquisition using color structured light and multi-pass dynamic programming , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[21]  Tobi Delbrück,et al.  Cooperative Stereo Matching Using Static and Dynamic Image Features , 1989, Analog VLSI Implementation of Neural Systems.

[22]  Jenny C. A. Read,et al.  A Bayesian Approach to the Stereo Correspondence Problem , 2002, Neural Computation.

[23]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[24]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Peter M. Will,et al.  Grid Coding: A Preprocessing Technique for Robot and Machine Vision , 1971, IJCAI.

[26]  Daniel Matolin,et al.  A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS , 2011, IEEE Journal of Solid-State Circuits.

[27]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  D. Nistér,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Eugenio Culurciello,et al.  Activity-driven, event-based vision sensors , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[31]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[32]  Tobi Delbrück,et al.  Asynchronous Event-Based Binocular Stereo Matching , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Takeo Kanade,et al.  The 3D Room: Digitizing Time-Varying 3D Events by Synchronized Multiple Video Streams , 1998 .

[34]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[35]  Szymon Rusinkiewicz,et al.  Spacetime Stereo: A Unifying Framework for Depth from Triangulation , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Christoph Sulzbachner,et al.  Bio-inspired Stereo Vision System with Silicon Retina Imagers , 2009, ICVS.

[37]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[38]  Andrew Gardner,et al.  Performance relighting and reflectance transformation with time-multiplexed illumination , 2005, ACM Trans. Graph..

[39]  Marc Pollefeys,et al.  Camera network calibration from dynamic silhouettes , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[40]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[41]  Ramesh Raskar,et al.  Fast separation of direct and global components of a scene using high frequency illumination , 2006, ACM Trans. Graph..

[42]  J I Nelson,et al.  Globality and stereoscopic fusion in binocular vision. , 1975, Journal of theoretical biology.

[43]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[44]  Zheng Zhi.  A Region Based Stereo Matching Algorithm Using Cooperative Optimization , 2009 .

[45]  Yuichi Taguchi,et al.  Stereo reconstruction with mixed pixels using adaptive over-segmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Jörg Kramer,et al.  An on/off transient imager with event-driven, asynchronous read-out , 2002, 2002 IEEE International Symposium on Circuits and Systems. Proceedings (Cat. No.02CH37353).