e-TLD: Event-Based Framework for Dynamic Object Tracking

This paper presents a long-term object tracking framework with a moving event camera under general tracking conditions. A first of its kind for these revolutionary cameras, the tracking framework uses a discriminative representation for the object with online learning, and detects and re-tracks the object when it comes back into the field-of-view. One of the key novelties is the use of an event-based local sliding window technique that tracks reliably in scenes with cluttered and textured background. In addition, Bayesian bootstrapping is used to assist real-time processing and boost the discriminative power of the object representation. On the other hand, when the object re-enters the field-of-view of the camera, a data-driven, global sliding window detector locates the object for subsequent tracking. Extensive experiments demonstrate the ability of the proposed framework to track and detect arbitrary objects of various shapes and sizes, including dynamic objects such as a human. This is a significant improvement compared to earlier works that simply track objects as long as they are visible under simpler background settings. Using the ground truth locations for five different objects under three motion settings, namely translation, rotation and 6-DOF, quantitative measurement is reported for the event-based tracking framework with critical insights on various performance issues. Finally, real-time implementation in C++ highlights tracking ability under scale, rotation, view-point and occlusion scenarios in a lab setting.

[1]  Bernabé Linares-Barranco,et al.  An Event-Driven Classifier for Spiking Neural Networks Fed with Synthetic or Dynamic Vision Sensor Data , 2017, Front. Neurosci..

[2]  Shih-Chii Liu,et al.  Effective sensor fusion with event-based sensors and deep network architectures , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[3]  Davide Scaramuzza,et al.  Low-latency visual odometry using event-based feature tracks , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[4]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[6]  Ryad Benosman,et al.  Simultaneous Mosaicing and Tracking with an Event Camera , 2014, BMVC.

[7]  Tobi Delbruck,et al.  Robotic goalie with 3 ms reaction time at 4% CPU load using event-based dynamic vision sensor , 2013, Front. Neurosci..

[8]  Tobi Delbrück,et al.  Combined frame- and event-based detection and tracking , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[9]  Yang Zhang,et al.  Iterative Multiple Hypothesis Tracking With Tracklet-Level Association , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Yiannis Aloimonos,et al.  Event-Based Moving Object Detection and Tracking , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[11]  Rui Jiang,et al.  Object tracking on event cameras with offline-online learning , 2020, CAAI Trans. Intell. Technol..

[12]  Tobi Delbrück,et al.  Training Deep Spiking Neural Networks Using Backpropagation , 2016, Front. Neurosci..

[13]  Qiang Wang,et al.  Fast Online Object Tracking and Segmentation: A Unifying Approach , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Ryad Benosman,et al.  Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual Features Tracking , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Davide Scaramuzza,et al.  EMVS: Event-based Multi-View Stereo , 2016, BMVC.

[16]  Masatoshi Ishikawa,et al.  Architectures and applications of high-speed vision , 2014 .

[17]  Klaus-Robert Müller,et al.  Incremental Support Vector Learning: Analysis, Implementation and Applications , 2006, J. Mach. Learn. Res..

[18]  Hong Yang,et al.  DART: Distribution Aware Retinal Transform for Event-Based Cameras , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Tobi Delbruck,et al.  Low Latency Event-Based Filtering and Feature Extraction for Dynamic Vision Sensors in Real-Time FPGA Applications , 2019, IEEE Access.

[20]  Stefan Leutenegger,et al.  Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera , 2016, ECCV.

[21]  Tobi Delbruck,et al.  A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor , 2014, IEEE Journal of Solid-State Circuits.

[22]  Arindam Basu,et al.  A Hybrid Neuromorphic Object Tracking and Classification Framework for Real-time Systems , 2020, IEEE transactions on neural networks and learning systems.

[23]  Chiara Bartolozzi,et al.  An Asynchronous Neuromorphic Event-Driven Visual Part-Based Shape Tracking , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Frédéric Jurie,et al.  Sampling Strategies for Bag-of-Features Image Classification , 2006, ECCV.

[25]  Yi Wu,et al.  Online Object Tracking: A Benchmark , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[27]  Tobi Delbrück,et al.  An embedded AER dynamic vision sensor for low-latency pole balancing , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[28]  Nitish V. Thakor,et al.  Real-time object recognition and orientation estimation using an event-based camera and CNN , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[29]  Arindam Basu,et al.  HyNNA: Improved Performance for Neuromorphic Vision Sensor Based Surveillance using Hybrid Neural Network Architecture , 2020, 2020 IEEE International Symposium on Circuits and Systems (ISCAS).

[30]  Davide Scaramuzza,et al.  Event-based, 6-DOF pose tracking for high-speed maneuvers , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[31]  Raul Cano.  On The Bayesian Bootstrap , 1992 .

[32]  Gregory Cohen,et al.  Event-Based Object Detection and Tracking for Space Situational Awareness , 2019, IEEE Sensors Journal.

[33]  Tobi Delbrück,et al.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM , 2016, Int. J. Robotics Res..

[34]  Shihao Zhang,et al.  Long-term object tracking with a moving event camera , 2018, BMVC.

[35]  Andrew Zisserman,et al.  Efficient additive kernels via explicit feature maps , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Davide Scaramuzza,et al.  EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time , 2017, IEEE Robotics and Automation Letters.

[37]  Alois Knoll,et al.  Online Multi-object Tracking-by-Clustering for Intelligent Transportation System with Neuromorphic Vision Sensor , 2017, KI.

[38]  Huosheng Hu,et al.  A Novel Real-Time Moving Target Tracking and Path Planning System for a Quadrotor UAV in Unknown Unstructured Outdoor Scenes , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[39]  Luping Shi,et al.  Robust Event-Based Object Tracking Combining Correlation Filter and CNN Representation , 2019, Front. Neurorobot..

[40]  Garrick Orchard,et al.  Spike context: A neuromorphic descriptor for pattern recognition , 2017, 2017 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[41]  A.N. Belbachir,et al.  Embedded Vision System for Real-Time Object Tracking using an Asynchronous Transient Vision Sensor , 2006, 2006 IEEE 12th Digital Signal Processing Workshop & 4th IEEE Signal Processing Education Workshop.