Event-Driven Sensing for Efficient Perception: Vision and audition algorithms

Event sensors implement circuits that capture partial functionality of biological sensors, such as the retina and cochlea. As with their biological counterparts, event sensors are drivers of their own output. That is, they produce dynamically sampled binary events to dynamically changing stimuli. Algorithms and networks that process this form of output representation are still in their infancy, but they show strong promise. This article illustrates the unique form of the data produced by the sensors and demonstrates how the properties of these sensor outputs make them useful for power-efficient, low-latency systems working in real time.

[1]  Ryan Kastner,et al.  $O(N)$O(N)-Space Spatiotemporal Filter for Reducing Noise in Neuromorphic Vision Sensors , 2021, IEEE Trans. Emerg. Top. Comput..

[2]  Chiara Bartolozzi,et al.  Robots with a sense of touch. , 2016, Nature materials.

[3]  Michael Pfeiffer,et al.  Deep Learning With Spiking Neurons: Opportunities and Challenges , 2018, Front. Neurosci..

[4]  Tobi Delbrück,et al.  Steering a predator robot using a mixed frame/event-driven convolutional neural network , 2016, 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP).

[5]  Kostas Daniilidis,et al.  EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras , 2018, Robotics: Science and Systems.

[6]  André van Schaik,et al.  Racing to learn: statistical inference and learning in a single spiking neuron with adaptive kernels , 2014, Front. Neurosci..

[7]  Garrick Orchard,et al.  Neuromorphic Event-Based 3D Pose Estimation , 2016, Front. Neurosci..

[8]  Adrian E. G. Huber,et al.  On Send-on-Delta sampling of bandlimited functions , 2017, 2017 International Conference on Sampling Theory and Applications (SampTA).

[9]  Aurel A. Lazar,et al.  Reconstructing Natural Visual Scenes From Spike Times , 2014, Proceedings of the IEEE.

[10]  Damien Querlioz,et al.  Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity , 2012, Neural Networks.

[11]  Marek Miskowicz,et al.  Send-On-Delta Concept: An Event-Based Data Reporting Strategy , 2006, Sensors (Basel, Switzerland).

[12]  Ryad Benosman,et al.  Complexity Analysis of Iterative Basis Transformations Applied to Event-Based Signals , 2018, Front. Neurosci..

[13]  G. Carter.  Coherence and time delay estimation , 1987, Proceedings of the IEEE.

[14]  Matthew Cook,et al.  Interacting maps for fast visual interpretation , 2011, The 2011 International Joint Conference on Neural Networks.

[15]  H. Landau.  Sampling, data transmission, and the Nyquist rate , 1967 .

[16]  Tobias Delbrück,et al.  Frame-free dynamic digital vision , 2008 .

[17]  Shih-Chii Liu,et al.  Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification , 2017, Front. Neurosci..

[18]  Tobi Delbruck,et al.  Evaluation of Event-Based Algorithms for Optical Flow with Ground-Truth from Inertial Measurement Sensor , 2016, Front. Neurosci..

[19]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[20]  Garrick Orchard,et al.  HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[22]  Zhe He,et al.  An event-driven probabilistic model of sound source localization using cochlea spikes , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[23]  Tobi Delbruck,et al.  Feature Representations for Neuromorphic Audio Spike Streams , 2018, Front. Neurosci..

[24]  Tobi Delbrück,et al.  Live demonstration: Convolutional neural network driven by dynamic vision sensor playing RoShamBo , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[25]  Kaushik Roy,et al.  Going Deeper in Spiking Neural Networks: VGG and Residual Architectures , 2018, Front. Neurosci..

[26]  Shih-Chii Liu,et al.  Neuromorphic sensory systems , 2010, Current Opinion in Neurobiology.

[27]  Davide Scaramuzza,et al.  Event-Based, 6-DOF Camera Tracking from Photometric Depth Maps , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Tobi Delbrück,et al.  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output , 2014, Proceedings of the IEEE.

[29]  Adam Osseiran,et al.  An Investigation into Spike-Based Neuromorphic Approaches for Artificial Olfactory Systems , 2017, Sensors.

[30]  Andre van Schaik,et al.  Asynchronous Binaural Spatial Audition Sensor With 2$\,\times\,$64$\,\times\,$4 Channel Output , 2014, IEEE Transactions on Biomedical Circuits and Systems.

[31]  Stefan Leutenegger,et al.  Simultaneous Optical Flow and Intensity Estimation from an Event Camera , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).