Toward a large-scale multimodal event-based dataset for neuromorphic deep learning applications

We discuss our efforts with event-based vision and describe our large-scale, heterogeneous robotic dataset that will add to the growing number of event-based datasets currently publicly available. Our dataset comprises over 10 hours of runtime from a mobile robot equipped with two DAVIS240C cameras and an Astra depth camera randomly wandering in an indoor environment while two other independently moving robots randomly wander in the same scene. Vicon ground truth pose is provided for all three robots. To our knowledge, this is the largest event-based dataset with ground truthed independently moving entities.

[1]  Tobi Delbrück,et al.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM , 2016, Int. J. Robotics Res..

[2]  Chiara Bartolozzi,et al.  Event-Based Visual Flow , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Roger Y. Tsai,et al.  A new technique for fully autonomous and efficient 3D robotics hand/eye calibration , 1988, IEEE Trans. Robotics Autom..

[4]  Chiara Bartolozzi,et al.  Independent motion detection with event-driven cameras , 2017, 2017 18th International Conference on Advanced Robotics (ICAR).

[5]  Yiu Cheung Shiu,et al.  Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX=XB , 1989, IEEE Trans. Robotics Autom..

[6]  Hannes Sommer,et al.  Evaluation of Combined Time-Offset Estimation and Hand-Eye Calibration on Robotic Datasets , 2017, FSR.

[7]  Yiannis Aloimonos,et al.  A Dataset for Visual Navigation with Neuromorphic Methods , 2016, Front. Neurosci..

[8]  Kostas Daniilidis,et al.  Hand-Eye Calibration Using Dual Quaternions , 1999, Int. J. Robotics Res..

[9]  Tobi Delbrück,et al.  Live demonstration: Convolutional neural network driven by dynamic vision sensor playing RoShamBo , 2017, 2017 IEEE International Symposium on Circuits and Systems (ISCAS).

[10]  Tobi Delbruck,et al.  Evaluation of Event-Based Algorithms for Optical Flow with Ground-Truth from Inertial Measurement Sensor , 2016, Front. Neurosci..

[11]  Davide Scaramuzza,et al.  Accurate Angular Velocity Estimation With an Event Camera , 2017, IEEE Robotics and Automation Letters.

[12]  Davide Scaramuzza,et al.  Real-time Visual-Inertial Odometry for Event Cameras using Keyframe-based Nonlinear Optimization , 2017, BMVC.

[13]  Tobi Delbrück,et al.  DDD17: End-To-End DAVIS Driving Dataset , 2017, ArXiv.

[14]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[15]  Davide Scaramuzza,et al.  EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time , 2017, IEEE Robotics and Automation Letters.

[16]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[17]  Roland Siegwart,et al.  Unified temporal and spatial calibration for multi-sensor systems , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[18]  Nicholas F. Y. Chen.  Pseudo-labels for supervised learning on event-based data , 2017, ArXiv.

[19]  Tobi Delbrück,et al.  Steering a predator robot using a mixed frame/event-driven convolutional neural network , 2016, 2016 Second International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP).

[20]  Nikolaos G. Tsagarakis,et al.  Real-Time Pose Estimation for Event Cameras with Stacked Spatial LSTM Networks , 2017, ArXiv.

[21]  Janet Wiles,et al.  Spike Event Based Learning in Neural Networks , 2015, ArXiv.

[22]  Davide Scaramuzza,et al.  Event-based, 6-DOF pose tracking for high-speed maneuvers , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[23]  Stefan Leutenegger,et al.  Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera , 2016, ECCV.

[24]  Torsten Bertram,et al.  Efficient trajectory optimization using a sparse model , 2013, 2013 European Conference on Mobile Robots.

[25]  O. Khatib,et al.  Real-Time Obstacle Avoidance for Manipulators and Mobile Robots , 1985, Proceedings. 1985 IEEE International Conference on Robotics and Automation.