Material Based Object Tracking in Hyperspectral Videos : Benchmark and Algorithms

Traditional color images only depict color intensities in red, green and blue channels, often making object trackers fail in challenging scenarios, e.g., background clutter and rapid changes of target appearance. Alternatively, material information of targets contained in a large amount of bands of hyperspectral images (HSI) is more robust to these difficult conditions. In this paper, we conduct a comprehensive study on how material information can be utilized to boost object tracking from three aspects: benchmark dataset, material feature representation and material based tracking. In terms of benchmark, we construct a dataset of fully-annotated videos, which contain both hyperspectral and color sequences of the same scene. Material information is represented by spectral-spatial histogram of multidimensional gradient, which describes the 3D local spectral-spatial structure in an HSI, and fractional abundances of constituted material components which encode the underlying material distribution. These two types of features are embedded into correlation filters, yielding material based tracking. Experimental results on the collected benchmark dataset show the potentials and advantages of material based object tracking.

[1]  Michael Felsberg,et al.  ECO: Efficient Convolution Operators for Tracking , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Song Wang,et al.  Learning Dynamic Siamese Network for Visual Object Tracking , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[3]  Derek Hoiem,et al.  Geometry-Informed Material Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Qingquan Li,et al.  Three-dimensional local binary patterns for hyperspectral imagery classification , 2017, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[5]  Ko Nishino,et al.  Automatically discovering local visual material attributes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Michael S. Brown,et al.  Do It Yourself Hyperspectral Imaging with Everyday Digital Cameras , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Qi Tian,et al.  Multi-cue Correlation Filters for Robust Visual Tracking , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Ming-Hsuan Yang,et al.  Learning Spatial-Aware Regressions for Visual Tracking , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Qingming Huang,et al.  Hedged Deep Tracking , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Jun Zhou,et al.  Object Tracking in Hyperspectral Videos with Convolutional Features and Kernelized Correlation Filter , 2018, ICSM.

[11]  Michael Felsberg,et al.  Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking , 2016, ECCV.

[12]  Rama Chellappa,et al.  Tracking via object reflectance using a hyperspectral video camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[13]  Jianke Zhu,et al.  A Scale Adaptive Kernel Correlation Filter Tracker with Feature Integration , 2014, ECCV Workshops.

[14]  Chein-I Chang,et al.  Real-Time Simplex Growing Algorithms for Hyperspectral Endmember Extraction , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Amit Banerjee,et al.  Hyperspectral video for illumination-invariant tracking , 2009, 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.

[16]  Ming-Hsuan Yang,et al.  Object Tracking Benchmark , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Yiannis Demiris,et al.  Context-Aware Deep Feature Compression for High-Speed Visual Tracking , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Yiannis Demiris,et al.  Visual Tracking Using Attention-Modulated Disintegration and Integration , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Jun Zhou,et al.  Hyperspectral Image Classification Based on Structured Sparse Logistic Regression and Three-Dimensional Wavelet Texture Features , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[20]  Li Zhang,et al.  Exploiting Spatial-Temporal Locality of Tracking via Structured Dictionary Learning , 2018, IEEE Transactions on Image Processing.

[21]  Michael Felsberg,et al.  Adaptive Color Attributes for Real-Time Visual Tracking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Simon Lucey,et al.  Learning Background-Aware Correlation Filters for Visual Tracking , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Bruce A. Draper,et al.  Visual object tracking using adaptive correlation filters , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Yuan Yan Tang,et al.  Dictionary Learning-Based Feature-Level Domain Adaptation for Cross-Scene Hyperspectral Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[25]  Suhad Lateef Al-khafaji,et al.  Spectral-Spatial Scale Invariant Feature Transform for Hyperspectral Images , 2018, IEEE Transactions on Image Processing.

[26]  Pong C. Yuen,et al.  Robust Visual Tracking via Basis Matching , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Ying Wang,et al.  Robust Hyperspectral Unmixing With Correntropy-Based Metric , 2013, IEEE Transactions on Image Processing.

[28]  Ming-Hsuan Yang,et al.  Hierarchical Convolutional Features for Visual Tracking , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[29]  Sim Heng Ong,et al.  A New Perspective on Material Classification and Ink Identification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Xiaogang Wang,et al.  Visual Tracking with Fully Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Dit-Yan Yeung,et al.  Learning a Deep Compact Image Representation for Visual Tracking , 2013, NIPS.

[32]  Tao Wang,et al.  Bio-Inspired Adaptive Hyperspectral Imaging for Real-Time Target Tracking , 2010, IEEE Sensors Journal.

[33]  Bin Wang,et al.  Material based salient object detection from hyperspectral images , 2018, Pattern Recognit..

[34]  Rui Caseiro,et al.  Exploiting the Circulant Structure of Tracking-by-Detection with Kernels , 2012, ECCV.

[35]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Qingshan Liu,et al.  Robust Visual Tracking via Convolutional Networks Without Training , 2015, IEEE Transactions on Image Processing.

[37]  Wolfgang Heidrich,et al.  Material Classification Using Raw Time-of-Flight Measurements , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Rob Heylen,et al.  Fully Constrained Least Squares Spectral Unmixing by Simplex Projection , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[39]  Xiaofeng Ren,et al.  Toward Robust Material Recognition for Everyday Objects , 2011, BMVC.

[40]  Antonio J. Plaza,et al.  Collaborative Sparse Regression for Hyperspectral Unmixing , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[41]  Jun Li,et al.  Spectral–Spatial Weighted Sparse Regression for Hyperspectral Image Unmixing , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[42]  Zhe Chen,et al.  An Experimental Survey on Correlation Filter-based Tracking , 2015, ArXiv.

[43]  Luca Bertinetto,et al.  End-to-End Representation Learning for Correlation Filter Based Tracking , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Bernard Ghanem,et al.  Context-Aware Correlation Filter Tracking , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Michael Felsberg,et al.  Discriminative Scale Space Tracking , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Anil Kokaram,et al.  The linear Monge-Kantorovitch linear colour mapping for example-based colour transfer , 2007 .

[47]  Yuan Yan Tang,et al.  Matrix-Vector Nonnegative Tensor Factorization for Blind Unmixing of Hyperspectral Imagery , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[48]  Arif Mahmood,et al.  Hyperspectral Face Recognition With Spatiospectral Information Fusion and PLS Regression , 2015, IEEE Transactions on Image Processing.

[49]  Ko Nishino,et al.  Visual Material Traits: Recognizing Per-Pixel Material Context , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[50]  Michael Felsberg,et al.  Convolutional Features for Correlation Filter Based Visual Tracking , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[51]  Takahiko Horiuchi,et al.  Spectral imaging method for material classification and inspection of printed circuit boards , 2010 .

[52]  Jiri Matas,et al.  Discriminative Correlation Filter with Channel and Spatial Reliability , 2017, CVPR.

[53]  Rui Caseiro,et al.  High-Speed Tracking with Kernelized Correlation Filters , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Yasuyuki Matsushita,et al.  High-resolution hyperspectral imaging via matrix factorization , 2011, CVPR 2011.

[55]  Zhe Chen,et al.  MUlti-Store Tracker (MUSTer): A cognitive psychology inspired approach to object tracking , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Matthew J. Hoffman,et al.  Integrating Hyperspectral Likelihoods in a Multidimensional Assignment Algorithm for Aerial Vehicle Tracking , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[57]  Michael Felsberg,et al.  Adaptive Decontamination of the Training Set: A Unified Formulation for Discriminative Visual Tracking , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Qing Guo,et al.  Dynamic Saliency-Aware Regularization for Correlation Filter-Based Object Tracking , 2019, IEEE Transactions on Image Processing.

[59]  Jocelyn Chanussot,et al.  Object Tracking by Hierarchical Decomposition of Hyperspectral Video Sequences: Application to Chemical Gas Plume Tracking , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[60]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[61]  Michael Felsberg,et al.  Learning Spatially Regularized Correlation Filters for Visual Tracking , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[62]  Émilie Chouzenoux,et al.  Fast Constrained Least Squares Spectral Unmixing Using Primal-Dual Interior-Point Optimization , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[63]  José M. Bioucas-Dias,et al.  Hyperspectral Subspace Identification , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[64]  José M. Bioucas-Dias,et al.  Vertex component analysis: a fast algorithm to unmix hyperspectral data , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[65]  Noah Snavely,et al.  Material recognition in the wild with the Materials in Context Database , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Subhransu Maji,et al.  Deep filter banks for texture recognition and segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Matthew J. Hoffman,et al.  Tracking in Aerial Hyperspectral Videos Using Deep Kernelized Correlation Filters , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[68]  Yasushi Yagi,et al.  Material Classification Using Frequency-and Depth-Dependent Time-of-Flight Distortion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Changsheng Xu,et al.  Deep Relative Tracking , 2017, IEEE Transactions on Image Processing.