Deep representation via convolutional neural network for classification of spatiotemporal event streams

Abstract Different from traditional frame-based cameras, event-based dynamic vision sensor (DVS) converts the visual information into spatiotemporal event streams. Convolutional neural networks (CNNs) have recently achieved outstanding classification performance while require a very large number of annotated samples. However, a lack of available large-scale event-stream datasets prevents application of CNNs to classification of such event streams. In this work, we show how the deep representation learned with an originally optimized CNN is efficiently transferred to the event-stream classification tasks. In our classification method, a spike-event temporal coding is used to encoding the spike-event information of each pixel. This temporal coding mechanism is implemented based on the subthreshold dynamic of the leaky integrate-and-fire (LIF) model. Three popular event-stream datasets were used to evaluate the performance of the proposed method. Results show that the proposed method leads to significantly improved classification accuracy, outperforming the current state of the art methods on the three event-stream datasets. Besides, the robustness of our method was verified in the MNIST-DVS dataset when Gaussian temporal noises were added to the timestamps of the events. Finally, we find that fine tuning with a small amount of event-stream data would improve the classification performance. This work can be easily extended to more complex scenarios and more fascinating and potential visual applications.