Neuromorophic vision sensing and processing

Conventional machine vision faces a fundamental latency-power tradeoff, where decreasing latency means increasing frame rate and power consumption. The communication by spikes in the brain provides the inspiration for developments of event-based vision sensors and processing, which could avoid this tradeoff entirely. This paper provides a personal perspective on developments of event-based vision sensors, algorithms, and applications over the period 2002–2016.

[1]  Ryad Benosman,et al.  Accelerated frame-free time-encoded multi-step imaging , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[2]  P. Lichtsteiner,et al.  64x64 Event-Driven Logarithmic Temporal Derivative Silicon Retina , 2005 .

[3]  Ryad Benosman,et al.  Asynchronous event-based corner detection and matching , 2015, Neural Networks.

[4]  Chiara Bartolozzi,et al.  Event-Based Visual Flow , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Tobi Delbrück,et al.  ELiSeD — An event-based line segment detector , 2016, 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP).

[6]  Tobias Delbrück,et al.  Frame-free dynamic digital vision , 2008 .

[7]  Tobi Delbrück,et al.  Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Heinrich Garn,et al.  Vehicle Counting with an Embedded Traffic Data System using an Optical Transient Sensor , 2007, 2007 IEEE Intelligent Transportation Systems Conference.

[9]  Tobi Delbruck,et al.  Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor , 2013, Front. Neurosci..

[10]  P. Lichtsteiner,et al.  Toward real-time particle tracking using an event-based dynamic vision sensor , 2011 .

[11]  Ania Mitros,et al.  Visual Sensor with Resolution Enhancement by Mechanical Vibrations , 2001, Auton. Robots.

[12]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[13]  Davide Scaramuzza,et al.  Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS) , 2016, 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP).

[14]  Chiara Bartolozzi,et al.  Asynchronous frameless event-based optical flow , 2012, Neural Networks.

[15]  Jörg Conradt,et al.  Autonomous indoor exploration with an event-based visual SLAM system , 2013, 2013 European Conference on Mobile Robots.

[16]  Tobi Delbrück,et al.  Fast sensory motor control based on event-based hybrid neuromorphic-procedural system , 2007, 2007 IEEE International Symposium on Circuits and Systems.

[17]  Tobi Delbrück,et al.  A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change , 2006, 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers.

[18]  Roland Siegwart,et al.  Adaptive pulsed laser line extraction for terrain reconstruction using a dynamic vision sensor , 2014, Front. Neurosci..

[19]  Tobi Delbruck,et al.  A 240×180 10mW 12us latency sparse-output vision sensor for mobile applications , 2013, 2013 Symposium on VLSI Circuits.

[20]  Bernabé Linares-Barranco,et al.  A 128$\,\times$ 128 1.5% Contrast Sensitivity 0.9% FPN 3 µs Latency 4 mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance Preamplifiers , 2013, IEEE Journal of Solid-State Circuits.

[21]  Eric R. Fossum,et al.  Active pixel sensors: are CCDs dinosaurs? , 1993, Electronic Imaging.

[22]  Tobias Brosch,et al.  Event-based optical flow on neuromorphic hardware , 2016, BICT.

[23]  R. Douglas,et al.  Event-Based Neuromorphic Systems , 2015 .

[24]  Ryad Benosman,et al.  Simultaneous Mosaicing and Tracking with an Event Camera , 2014, BMVC.

[25]  Ryad Benosman,et al.  Event-based features for robotic vision , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[26]  Tobias Brosch,et al.  On event-based optical flow detection , 2015, Front. Neurosci..

[27]  Shih-Chii Liu,et al.  Neuromorphic sensory systems , 2010, Current Opinion in Neurobiology.

[28]  Bernabé Linares-Barranco,et al.  Fast Vision Through Frameless Event-Based Sensing and Convolutional Processing: Application to Texture Recognition , 2010, IEEE Transactions on Neural Networks.

[29]  J. Kim,et al.  Dynamic vision sensor for low power applications , 2014, The 18th IEEE International Symposium on Consumer Electronics (ISCE 2014).

[30]  Bernabé Linares-Barranco,et al.  An Event-Driven Multi-Kernel Convolution Processor Module for Event-Driven Vision Sensors , 2012, IEEE Journal of Solid-State Circuits.

[31]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[32]  Tobi Delbruck.  Fun with asynchronous vision sensors and processing , 2012 .

[33]  David Johannes Borer,et al.  4D Flow Visualization with Dynamic Vision Sensors , 2014 .

[34]  Tobi Delbrück,et al.  Live demonstration: A pencil balancing robot using a pair of AER dynamic vision sensors , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[35]  Bernabé Linares-Barranco,et al.  A 3.6 $\mu$ s Latency Asynchronous Frame-Free Event-Driven Dynamic-Vision-Sensor , 2011, IEEE Journal of Solid-State Circuits.

[36]  Davide Scaramuzza,et al.  Event-based, 6-DOF pose tracking for high-speed maneuvers , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[37]  Tobi Delbrück,et al.  Steering a predator robot using a mixed frame/event-driven convolutional neural network , 2016, 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP).

[38]  Massimo A. Sivilotti,et al.  Wiring considerations in analog VLSI systems, with application to field-programmable networks , 1992 .

[39]  Zhen Wang,et al.  Draining our glass: an energy and heat characterization of Google Glass , 2014, APSys.

[40]  Tobi Delbrück,et al.  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output , 2014, Proceedings of the IEEE.

[41]  Daniel Matolin,et al.  A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS , 2011, IEEE Journal of Solid-State Circuits.

[42]  Tobi Delbrück,et al.  Asynchronous Event-Based Binocular Stereo Matching , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[43]  Tobi Delbrück,et al.  Human vs. computer slot car racing using an event and frame-based DAVIS vision sensor , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[44]  Tobi Delbruck,et al.  Evaluation of Event-Based Algorithms for Optical Flow with Ground-Truth from Inertial Measurement Sensor , 2016, Front. Neurosci..

[45]  Luca Benini,et al.  Origami: A Convolutional Network Accelerator , 2015, ACM Great Lakes Symposium on VLSI.

[46]  Tobi Delbrück,et al.  Design of an RGBW color VGA rolling and global shutter dynamic and active-pixel vision sensor , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[47]  Tobi Delbrück,et al.  CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware Sensory–Processing– Learning–Actuating System for High-Speed Visual Object Recognition and Tracking , 2009, IEEE Transactions on Neural Networks.

[48]  Tobi Delbrück,et al.  Integration of dynamic vision sensor with inertial measurement unit for electronically stabilized event-based vision , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[49]  John Wawrzynek,et al.  Silicon Auditory Processors as Computer Peripherals , 1992, NIPS.

[50]  Stephan Schraml,et al.  CARE: A dynamic stereo vision sensor system for fall detection , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[51]  Misha Anne Mahowald,et al.  VLSI analogs of neuronal visual processing: a synthesis of form and function , 1992 .

[52]  Davide Scaramuzza,et al.  Lifetime estimation of events from Dynamic Vision Sensors , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[53]  Stephan Schraml,et al.  Event-driven stereo vision for fall detection , 2011, CVPR 2011 WORKSHOPS.

[54]  Tobi Delbrück,et al.  A pencil balancing robot using a pair of AER dynamic vision sensors , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[55]  Stefan Leutenegger,et al.  Simultaneous Optical Flow and Intensity Estimation from an Event Camera , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Daniel Matolin,et al.  True correlated double sampling and comparator design for time-based image sensors , 2009, 2009 IEEE International Symposium on Circuits and Systems.

[57]  Tobi Delbrück,et al.  Addressable current reference array with 170dB dynamic range , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[58]  Bernabé Linares-Barranco,et al.  Advanced Vision Processing Systems: Spike-Based Simulation and Processing , 2009, ACIVS.

[59]  Tobi Delbrück,et al.  Real-time, high-speed video decompression using a frame- and event-based DAVIS sensor , 2014, 2014 IEEE International Symposium on Circuits and Systems (ISCAS).

[60]  Tobi Delbruck,et al.  A Dynamic Vision Sensor With 1% Temporal Contrast Sensitivity and In-Pixel Asynchronous Delta Modulator for Event Encoding , 2015, IEEE Journal of Solid-State Circuits.

[61]  Tobi Delbrück,et al.  Live demonstration: Gesture-based remote control using stereo pair of dynamic vision sensors , 2012, 2012 IEEE International Symposium on Circuits and Systems.

[62]  Bernabé Linares-Barranco,et al.  A 32$\,\times\,$ 32 Pixel Convolution Processor Chip for Address Event Vision Sensors With 155 ns Event Latency and 20 Meps Throughput , 2011, IEEE Transactions on Circuits and Systems I: Regular Papers.

[63]  Kwabena Boahen,et al.  A burst-mode word-serial address-event link-I: transmitter design , 2004, IEEE Transactions on Circuits and Systems I: Regular Papers.

[64]  Jongkil Park,et al.  Improved contrast sensitivity DVS and its application to event-driven stereo vision , 2013, 2013 IEEE International Symposium on Circuits and Systems (ISCAS2013).