Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception

As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution. It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams. Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed. In this tutorial-like article, a comprehensive review of the emerging technology is given. First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced. The signal processing techniques for event noise processing and event data representation are then discussed. Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed. Finally, challenges and future research directions are pointed out. It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities.

[1]  Shih-Chii Liu,et al.  Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification , 2017, Front. Neurosci..

[2]  Kostas Daniilidis,et al.  EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras , 2018, Robotics: Science and Systems.

[3]  Ryad Benosman,et al.  A sparse coding multi-scale precise-timing machine learning algorithm for neuromorphic event-based sensors , 2018, Defense + Security.

[4]  Kostas Daniilidis,et al.  Unsupervised Event-Based Learning of Optical Flow, Depth, and Egomotion , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Alberto Ferreira de Souza,et al.  Self-Driving Cars: A Survey , 2019, Expert Syst. Appl..

[6]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[7]  Emilio Frazzoli,et al.  Wormhole Learning , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[8]  Alois Knoll,et al.  EDDD: Event-Based Drowsiness Driving Detection Through Facial Motion Analysis With Neuromorphic Vision Sensor , 2020, IEEE Sensors Journal.

[9]  Alois Knoll,et al.  Multi-Cue Event Information Fusion for Pedestrian Detection With Neuromorphic Vision Sensors , 2019, Front. Neurorobot..

[10]  William Whittaker,et al.  Autonomous driving in urban environments: Boss and the Urban Challenge , 2008, J. Field Robotics.

[11]  Matthew Cook,et al.  Interacting maps for fast visual interpretation , 2011, The 2011 International Joint Conference on Neural Networks.

[12]  Tobi Delbruck,et al.  Event-Driven Sensing for Efficient Perception: Vision and audition algorithms , 2019, IEEE Signal Processing Magazine.

[13]  Deepak Khosla,et al.  Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition , 2014, International Journal of Computer Vision.

[14]  Tobi Delbrück,et al.  Design of a spatiotemporal correlation filter for event-based sensors , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[15]  Davide Scaramuzza,et al.  End-to-End Learning of Representations for Asynchronous Event-Based Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  C. Mead,et al.  Neuromorphic Robot Vision with Mixed Analog- Digital Architecture , 2005 .

[17]  Luping Shi,et al.  Robust Event-Based Object Tracking Combining Correlation Filter and CNN Representation , 2019, Front. Neurorobot..

[18]  Nick Barnes,et al.  Continuous-time Intensity Estimation Using Event Cameras , 2018, ACCV.

[19]  Hang Su,et al.  Neuromorphic Visual Odometry System For Intelligent Vehicle Application With Bio-inspired Vision Sensor , 2019, 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[20]  Nitish V. Thakor,et al.  HFirst: A Temporal Approach to Object Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Vladlen Koltun,et al.  Events-To-Video: Bringing Modern Computer Vision to Event Cameras , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Arindam Basu,et al.  Spiking Neural Network Based Region Proposal Networks for Neuromorphic Vision Sensors , 2019, 2019 IEEE International Symposium on Circuits and Systems (ISCAS).

[23]  Ryad Benosman,et al.  HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Horst Bischof,et al.  Event-driven stereo matching for real-time 3D panoramic vision , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Tiejun Huang,et al.  Event-Based Vision Enhanced: A Joint Detection Framework in Autonomous Driving , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[26]  Vijay Kumar,et al.  The Multivehicle Stereo Event Camera Dataset: An Event Camera Dataset for 3D Perception , 2018, IEEE Robotics and Automation Letters.

[27]  Chiara Bartolozzi,et al.  Event-Based Vision: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Wei Li,et al.  DET: A High-Resolution DVS Dataset for Lane Extraction , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Garrick Orchard,et al.  An Event-Driven Categorization Model for AER Image Sensors Using Multispike Encoding and Learning , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Jörg Conradt,et al.  FLGR: Fixed Length Gists Representation Learning for RNN-HMM Hybrid-Based Neuromorphic Continuous Gesture Recognition , 2019, Front. Neurosci..

[31]  Vijayan K. Asari,et al.  Inceptive Event Time-Surfaces for Object Classification Using Neuromorphic Cameras , 2019, ICIAR.

[32]  Nicholas F. Y. Chen.  Pseudo-Labels for Supervised Learning on Dynamic Vision Sensor Data, Applied to Object Detection Under Ego-Motion , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Tobi Delbrück,et al.  Training Deep Spiking Neural Networks Using Backpropagation , 2016, Front. Neurosci..

[34]  Ana Cristina Murillo,et al.  EV-SegNet: Semantic Segmentation for Event-Based Cameras , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  Ralph Etienne-Cummings,et al.  Towards a Cortical Prosthesis: Implementing A Spike-Based HMAX Model of Visual Object Recognition in Silico , 2011, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[36]  Stephan Schraml,et al.  Spatiotemporal multiple persons tracking using Dynamic Vision Sensor , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[37]  Chiara Bartolozzi,et al.  Event-Based Visual Flow , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[38]  Tiejun Huang,et al.  Spike Coding for Dynamic Vision Sensor in Intelligent Driving , 2019, IEEE Internet of Things Journal.

[39]  Davide Scaramuzza,et al.  ESIM: an Open Event Camera Simulator , 2018, CoRL.

[40]  Thomas Pock,et al.  Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation , 2016, International Journal of Computer Vision.

[41]  Yiannis Aloimonos,et al.  Unsupervised Learning of Dense Optical Flow and Depth from Sparse Event Data , 2018, ArXiv.

[42]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[43]  Garrick Orchard,et al.  A Noise Filtering Algorithm for Event-Based Asynchronous Change Detection Image Sensors on TrueNorth and Its Implementation on TrueNorth , 2018, Front. Neurosci..

[44]  Narciso García,et al.  Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Ryan Kastner,et al.  $O(N)$O(N)-Space Spatiotemporal Filter for Reducing Noise in Neuromorphic Vision Sensors , 2021, IEEE Trans. Emerg. Top. Comput..

[46]  Michael Pfeiffer,et al.  Deep Learning With Spiking Neurons: Opportunities and Challenges , 2018, Front. Neurosci..

[47]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[48]  Garrick Orchard,et al.  HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Emilio Frazzoli,et al.  Cross-Modal Learning Filters for RGB-Neuromorphic Wormhole Learning , 2019, Robotics: Science and Systems.

[50]  Tobi Delbrück,et al.  DDD17: End-To-End DAVIS Driving Dataset , 2017, ArXiv.

[51]  Kaushik Roy,et al.  Going Deeper in Spiking Neural Networks: VGG and Residual Architectures , 2018, Front. Neurosci..

[52]  Bowen Du,et al.  EV-Gait: Event-Based Robust Gait Recognition Using Dynamic Vision Sensors , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Tobi Delbrück,et al.  Retinomorphic Event-Based Vision Sensors: Bioinspired Cameras With Spiking Output , 2014, Proceedings of the IEEE.