Energy-Efficient Spiking Segmenter for Frame and Event-Based Images

Semantic segmentation predicts dense pixel-wise semantic labels, which is crucial for autonomous environment perception systems. For applications on mobile devices, current research focuses on energy-efficient segmenters for both frame and event-based cameras. However, there is currently no artificial neural network (ANN) that can perform efficient segmentation on both types of images. This paper introduces spiking neural network (SNN, a bionic model that is energy-efficient when implemented on neuromorphic hardware) and develops a Spiking Context Guided Network (Spiking CGNet) with substantially lower energy consumption and comparable performance for both frame and event-based images. First, this paper proposes a spiking context guided block that can extract local features and context information with spike computations. On this basis, the directly-trained SCGNet-S and SCGNet-L are established for both frame and event-based images. Our method is verified on the frame-based dataset Cityscapes and the event-based dataset DDD17. On the Cityscapes dataset, SCGNet-S achieves comparable results to ANN CGNet with 4.85 × energy efficiency. On the DDD17 dataset, Spiking CGNet outperforms other spiking segmenters by a large margin.

[1]  T. Masquelier,et al.  Optical flow estimation from event-based cameras and spiking neural networks , 2023, Frontiers in Neuroscience.

[2]  Zhaofei Yu,et al.  Reducing ANN-SNN Conversion Error through Residual Membrane Potential , 2023, AAAI.

[3]  Yisa Zhang,et al.  Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network , 2023, Micromachines.

[4]  Yu Zhang,et al.  Isomorphic model-based initialization for convolutional neural networks , 2022, J. Vis. Commun. Image Represent..

[5]  Yaowei Wang,et al.  Spikformer: When Spiking Neural Network Meets Transformer , 2022, ICLR.

[6]  Lei Deng,et al.  Attention Spiking Neural Networks , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Jiacong Xu,et al.  PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Benoît Miramond,et al.  Object Detection with Spiking Neural Networks on Automotive Event Data , 2022, 2022 International Joint Conference on Neural Networks (IJCNN).

[9]  Yi Zeng,et al.  Efficient and Accurate Conversion of Spiking Neural Network with Burst Spikes , 2022, IJCAI.

[10]  D. Scaramuzza,et al.  ESS: Learning Event-based Semantic Segmentation from Still Images , 2022, ECCV.

[11]  Zhaofei Yu,et al.  Optimized Potential Initialization for Low-latency Spiking Neural Networks , 2022, AAAI Conference on Artificial Intelligence.

[12]  Elisabetta Chicca,et al.  Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision , 2021, GCPR.

[13]  A. Schwing,et al.  Masked-attention Mask Transformer for Universal Image Segmentation , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Priyadarshini Panda,et al.  Beyond classification: directly training spiking neural networks for semantic segmentation , 2021, Neuromorph. Comput. Eng..

[15]  Kuk-Jin Yoon,et al.  Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Guoqi Li,et al.  Temporal-wise Attention Spiking Neural Networks for Event Streams Classification , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Silvia Tolu,et al.  RetinaNet Object Detector Based on Analog-to-Spiking Neural Network Conversion , 2021, 2021 8th International Conference on Soft Computing & Machine Intelligence (ISCMI).

[18]  Anima Anandkumar,et al.  SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers , 2021, NeurIPS.

[19]  Cordelia Schmid,et al.  Segmenter: Transformer for Semantic Segmentation , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Saibal Mukhopadhyay,et al.  A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection , 2021, IEEE Transactions on Image Processing.

[21]  Shi Gu,et al.  Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks , 2021, ICLR.

[22]  Tiejun Huang,et al.  Deep Residual Learning in Spiking Neural Networks , 2021, NeurIPS.

[23]  Weichao Sun,et al.  Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes , 2021, ArXiv.

[24]  Tal Hassner,et al.  HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Michael Ying Yang,et al.  CABiNet: Efficient Context Aggregation Network for Low-Latency Semantic Segmentation , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Lei Deng,et al.  Going Deeper With Directly-Trained Larger Spiking Neural Networks , 2020, AAAI.

[27]  Kai Huang,et al.  Target Tracking Control of a Wheel-less Snake Robot Based on a Supervised Multi-layered SNN , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  P. Panda,et al.  Revisiting Batch Normalization for Training Low-Latency Deep Spiking Neural Networks From Scratch , 2020, Frontiers in Neuroscience.

[29]  Yonghong Tian,et al.  Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  Gang Yu,et al.  BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic Segmentation , 2020, International Journal of Computer Vision.

[31]  Min Xu,et al.  SiamSNN: Siamese Spiking Neural Networks for Energy-Efficient Object Tracking , 2020, ICANN.

[32]  Kostas Daniilidis,et al.  Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks , 2020, ECCV.

[33]  Davide Scaramuzza,et al.  Video to Events: Recycling Video Datasets for Event Cameras , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Kaushik Roy,et al.  Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures , 2019, Frontiers in Neuroscience.

[35]  Sungroh Yoon,et al.  Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection , 2019, AAAI.

[36]  Emre Neftci,et al.  Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-based optimization to spiking neural networks , 2019, IEEE Signal Processing Magazine.

[37]  Ana Cristina Murillo,et al.  EV-SegNet: Semantic Segmentation for Event-Based Cameras , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[38]  Sheng Tang,et al.  CGNet: A Light-Weight Context Guided Network for Semantic Segmentation , 2018, IEEE Transactions on Image Processing.

[39]  Garrick Orchard,et al.  SLAYER: Spike Layer Error Reassignment in Time , 2018, NeurIPS.

[40]  Gang Yu,et al.  BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation , 2018, ECCV.

[41]  Linda G. Shapiro,et al.  ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation , 2018, ECCV.

[42]  Kaushik Roy,et al.  Going Deeper in Spiking Neural Networks: VGG and Residual Architectures , 2018, Front. Neurosci..

[43]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[44]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[45]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[46]  Luis Miguel Bergasa,et al.  Efficient ConvNet for real-time semantic segmentation , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[47]  Lei Deng,et al.  Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks , 2017, Front. Neurosci..

[48]  Tobi Delbrück,et al.  DDD17: End-To-End DAVIS Driving Dataset , 2017, ArXiv.

[49]  Iulia-Alexandra Lungu,et al.  Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networks , 2016, ArXiv.

[50]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Bastian Leibe,et al.  Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Ian D. Reid,et al.  RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Eugenio Culurciello,et al.  ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation , 2016, ArXiv.

[55]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[58]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[60]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[61]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[63]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[64]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[66]  Lei Deng,et al.  Advancing Residual Learning towards Powerful Deep Spiking Neural Networks , 2021, ArXiv.

[67]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).