Time-Aggregation-Based Lossless Video Encoding for Neuromorphic Vision Sensor Data

Dynamic vision sensors (DVSs) are emerging neuromorphic visual capturing devices, with great advantages in terms of low-power consumption, wide dynamic range, and high temporal resolution in diverse applications, such as autonomous driving, robotics, tactile sensing, and drones. The capturing method results in lower data rates than conventional video. Still, such data can be further compressed. Recent research has shown great benefits of temporal data aggregation on event-based vision data utilization. According to recent results, time aggregation of DVS data not only reduces the data rate but improves classification and object detection accuracy. In this work, we propose a compression strategy, time-aggregation-based lossless video encoding for neuromorphic vision sensor data (TALVEN), which utilizes temporal data aggregation, arrangement of the data in a specific format and lossless video encoding techniques to achieve high compression ratios. The detailed experimental analysis on outdoor and indoor datasets shows that our proposed strategy achieves superior compression ratios than the best state-of-the-art strategies.

[1]  Maria G. Martini,et al.  Performance Comparison of Lossless Compression Strategies for Dynamic Vision Sensor Data , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Maria G. Martini,et al.  Lossless Compression of Data From Static and Mobile Dynamic Vision Sensors-Performance and Trade-Offs , 2020, IEEE Access.

[3]  Davide Scaramuzza,et al.  EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time , 2017, International Journal of Computer Vision.

[4]  Tobi Delbruck,et al.  Event-Driven Sensing for Efficient Perception: Vision and audition algorithms , 2019, IEEE Signal Processing Magazine.

[5]  Jyrki Alakuijala,et al.  Brotli Compressed Data Format , 2016, RFC.

[6]  Ryad Benosman,et al.  Spatial and Temporal Downsampling in Event-Based Visual Classification , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Maria G. Martini,et al.  Bandwidth Modeling of Silicon Retinas for Next Generation Visual Sensor Networks , 2019, Sensors.

[8]  Maria G. Martini,et al.  Data rate estimation based on scene complexity for dynamic vision sensors on unmanned vehicles , 2018, 2018 IEEE 29th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC).

[9]  Min Liu,et al.  Adaptive Time-Slice Block-Matching Optical Flow Algorithm for Dynamic Vision Sensors , 2018, BMVC.

[10]  Tobi Delbrück,et al.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM , 2016, Int. J. Robotics Res..

[11]  T. Delbruck,et al.  A 128 128 120 dB 15 s Latency Asynchronous Temporal Contrast Vision Sensor , 2006 .

[12]  Leonid Boytsov,et al.  Decoding billions of integers per second through vectorization , 2012, Softw. Pract. Exp..

[13]  Tiejun Huang,et al.  Spike Coding for Dynamic Vision Sensor in Intelligent Driving , 2019, IEEE Internet of Things Journal.

[14]  Narciso García,et al.  Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Tobi Delbrück,et al.  A 128 X 128 120db 30mw asynchronous vision sensor that responds to relative intensity change , 2006, 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers.

[16]  Matteo Matteucci,et al.  Asynchronous Convolutional Networks for Object Detection in Neuromorphic Cameras , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  Mohammad Shikh-Bahaei,et al.  Challenges and Perspectives in Neuromorphic-based Visual IoT Systems and Networks , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Fariborz Baghaei Naeini,et al.  A Novel Dynamic-Vision-Based Approach for Tactile Sensing Applications , 2020, IEEE Transactions on Instrumentation and Measurement.

[19]  Davide Scaramuzza,et al.  Event-based, 6-DOF pose tracking for high-speed maneuvers , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Tiejun Huang,et al.  Spike Coding for Dynamic Vision Sensors , 2018, 2018 Data Compression Conference.

[21]  Davide Scaramuzza,et al.  Feature detection and tracking with the dynamic and active-pixel vision sensor (DAVIS) , 2016, 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP).

[22]  Chiara Bartolozzi,et al.  Event-Based Vision: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Heinrich Garn,et al.  Estimation of Vehicle Speed Based on Asynchronous Data from a Silicon Retina Optical Sensor , 2006, 2006 IEEE Intelligent Transportation Systems Conference.

[24]  Peter Deutsch,et al.  ZLIB Compressed Data Format Specification version 3.3 , 1996, RFC.

[25]  Tiejun Huang,et al.  Event-Based Vision Enhanced: A Joint Detection Framework in Autonomous Driving , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[26]  Fariborz Baghaei Naeini,et al.  A Novel Event-Based Incipient Slip Detection Using Dynamic Active-Pixel Vision Sensor (DAVIS) , 2018, Sensors.

[27]  Garrick Orchard,et al.  HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.