DS-KCF: a real-time tracker for RGB-D data

We propose an RGB-D single-object tracker, built upon the extremely fast RGB-only KCF tracker that is able to exploit depth information to handle scale changes, occlusions, and shape changes. Despite the computational demands of the extra functionalities, we still achieve real-time performance rates of 35–43 fps in MATLAB and 187 fps in our C++ implementation. Our proposed method includes fast depth-based target object segmentation that enables, (1) efficient scale change handling within the KCF core functionality in the Fourier domain, (2) the detection of occlusions by temporal analysis of the target’s depth distribution, and (3) the estimation of a target’s change of shape through the temporal evolution of its segmented silhouette allows. Finally, we provide an in-depth analysis of the factors affecting the throughput and precision of our proposed tracker and perform extensive comparative analysis. Both the MATLAB and C++ versions of our software are available in the public domain.

[1]  Zhe Chen,et al.  An Experimental Survey on Correlation Filter-based Tracking , 2015, ArXiv.

[2]  Shao-Yi Chien,et al.  Fast image segmentation based on K-Means clustering with histograms in HSV color space , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[3]  Michael Felsberg,et al.  Adaptive Color Attributes for Real-Time Visual Tracking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Simon Lucey,et al.  Multi-channel Correlation Filters , 2013, 2013 IEEE International Conference on Computer Vision.

[5]  Qi Wang,et al.  Multi-cue based tracking , 2014, Neurocomputing.

[6]  Luis Salgado,et al.  Depth-Color Fusion Strategy for 3-D Scene Modeling With Kinect , 2013, IEEE Transactions on Cybernetics.

[7]  Massimo Piccardi,et al.  Local Depth Patterns for Tracking in Depth Videos , 2015, ACM Multimedia.

[8]  Benxiong Huang,et al.  Channel Capacity Analysis of the Multiple Orthogonal Sequence Spread Spectrum Watermarking in Audio Signals , 2016, IEEE Signal Processing Letters.

[9]  David Zhang,et al.  Fast Visual Tracking via Dense Spatio-temporal Context Learning , 2014, ECCV.

[10]  V. Beran,et al.  Depth-Based Filtration for Tracking Boost , 2015, ACIVS.

[11]  Gang Wang,et al.  Real-time part-based visual tracking via adaptive correlation filters , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Shai Avidan,et al.  Support vector tracking , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Ming-Hsuan Yang,et al.  Long-term correlation tracking , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Yi Wu,et al.  Online Object Tracking: A Benchmark , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Jitendra Malik,et al.  Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Jitendra Malik,et al.  Learning Rich Features from RGB-D Images for Object Detection and Segmentation , 2014, ECCV.

[17]  Majid Mirmehdi,et al.  Multiple Human Tracking in RGB-D Data: A Survey , 2016, ArXiv.

[18]  Stefan Roth,et al.  MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking , 2015, ArXiv.

[19]  Junseok Kwon,et al.  Visual tracking decomposition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Armin B. Cremers,et al.  Boosting scalable gradient features for adaptive real-time tracking , 2011, 2011 IEEE International Conference on Robotics and Automation.

[21]  Rui Caseiro,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence High-speed Tracking with Kernelized Correlation Filters , 2022 .

[22]  Majid Mirmehdi,et al.  Multiple human tracking in RGB-depth data: a survey , 2017, IET Comput. Vis..

[23]  Hao Liu,et al.  A rotation adaptive correlation filter for robust tracking , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[24]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.

[25]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[26]  Wilhelm Burger,et al.  Digital Image Processing - An Algorithmic Introduction using Java , 2008, Texts in Computer Science.

[27]  Hanqing Lu,et al.  Collaborative Correlation Tracking , 2015, BMVC.

[28]  Lei Zhang,et al.  Real-Time Compressive Tracking , 2012, ECCV.

[29]  Simone Calderara,et al.  Visual Tracking: An Experimental Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  D Casasent,et al.  Multivariant technique for multiclass pattern recognition. , 1980, Applied optics.

[31]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Xiangmin Xu,et al.  Multi-invariance appearance model for object tracking , 2015, 2015 IEEE International Conference on Digital Signal Processing (DSP).

[33]  Shin Ishii,et al.  An occlusion-aware particle filter tracker to handle complex and persistent occlusions , 2016, Computer Vision and Image Understanding.

[34]  Vibhav Vineet,et al.  Struck: Structured Output Tracking with Kernels , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Michael Felsberg,et al.  Accurate Scale Estimation for Robust Visual Tracking , 2014, BMVC.

[36]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Hang Li,et al.  Patch-based Scale Calculation for Real-time Visual Tracking , 2016, IEEE Signal Processing Letters.

[38]  Jianke Zhu,et al.  A Scale Adaptive Kernel Correlation Filter Tracker with Feature Integration , 2014, ECCV Workshops.

[39]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[40]  Jianxiong Xiao,et al.  Tracking Revisited Using RGBD Camera: Unified Benchmark and Baselines , 2013, 2013 IEEE International Conference on Computer Vision.

[41]  Michael Beetz,et al.  Depth-adaptive superpixels , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[42]  Yang Li,et al.  Reliable Patch Trackers: Robust visual tracking by exploiting reliable patches , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Bruce A. Draper,et al.  Visual object tracking using adaptive correlation filters , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[45]  Armin B. Cremers,et al.  Adaptive Multi-cue 3D Tracking of Arbitrary Objects , 2012, DAGM/OAGM Symposium.

[46]  Majid Mirmehdi,et al.  Real-time RGB-D Tracking with Depth Scaling Kernelised Correlation Filters and Occlusion Handling , 2015, BMVC.

[47]  Zhongfei Zhang,et al.  A survey of appearance models in visual object tracking , 2013, ACM Trans. Intell. Syst. Technol..

[48]  Luc Van Gool,et al.  Beyond semi-supervised tracking: Tracking should be as simple as detection, but not simpler than recognition , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.