GCNNMatch: Graph Convolutional Neural Networks for Multi-Object Tracking via Sinkhorn Normalization

This paper proposes a novel method for online Multi-Object Tracking (MOT) using Graph Convolutional Neural Network (GCNN) based feature extraction and end-to-end feature matching for object association. The Graph based approach incorporates both appearance and geometry of objects at past frames as well as the current frame into the task of feature learning. This new paradigm enables the network to leverage the "context" information of the geometry of objects and allows us to model the interactions among the features of multiple objects. Another central innovation of our proposed framework is the use of the Sinkhorn algorithm for end-to-end learning of the associations among objects during model training. The network is trained to predict object associations by taking into account constraints specific to the MOT task. Experimental results demonstrate the efficacy of the proposed approach in achieving top performance on the MOT16 & 17 Challenge problems among state-of-the-art online and supervised approaches. The code is available at this https URL.

[1]  Zhang Xiong,et al.  Multiplex Labeling Graph for Near-Online Tracking in Crowded Scenes , 2020, IEEE Internet of Things Journal.

[2]  Bin Liu,et al.  GSM: Graph Similarity Model for Multi-Object Tracking , 2020, IJCAI.

[3]  Kris Kitani,et al.  GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking With 2D-3D Multi-Feature Learning , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Bin Liu,et al.  DASOT: A Unified Framework Integrating Data Association and Single Object Tracking for Online Multi-Object Tracking , 2020, AAAI.

[5]  Jiahe Li,et al.  Graph Networks for Multiple Object Tracking , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[6]  Jeonghwan Gwak,et al.  OneShotDA: Online Multi-Object Tracker With One-Shot-Learning-Based Data Association , 2020, IEEE Access.

[7]  Margret Keuper,et al.  Unsupervised Multiple Person Tracking using AutoEncoder-Based Lifted Multicuts , 2020, ArXiv.

[8]  L. Leal-Taix'e,et al.  Learning a Neural Solver for Multiple Object Tracking , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[10]  Tomasz Malisiewicz,et al.  SuperGlue: Learning Feature Matching With Graph Neural Networks , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Yong Yan,et al.  Multi-Target Tracking with Trajectory Prediction and Re-Identification , 2019, 2019 Chinese Automation Congress (CAC).

[12]  Shengjin Wang,et al.  Towards Real-Time Multi-Object Tracking , 2019, ECCV.

[13]  Francisco Herrera,et al.  Deep Learning in Video Multi-Object Tracking: A Survey , 2019, Neurocomputing.

[14]  Xiantong Zhen,et al.  Graph Neural Based End-to-end Data Association Framework for Online Multiple-Object Tracking , 2019, ArXiv.

[15]  R. Horaud,et al.  How to Train Your Deep Multi-Object Tracker , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Fan Yang,et al.  Deep Association: End-to-end Graph-Based Learning for Multiple Object Tracking with Conv-Graph Neural Network , 2019, ICMR.

[17]  Haibin Ling,et al.  FAMNet: Joint Learning of Feature, Affinity and Multi-Dimensional Assignment for Online Multiple Object Tracking , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[18]  Laura Leal-Taixé,et al.  Tracking Without Bells and Whistles , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[20]  Jonathon A. Chambers,et al.  Multi-Level Cooperative Fusion of GM-PHD Filters for Online Multiple Human Tracking , 2019, IEEE Transactions on Multimedia.

[21]  Kwangjin Yoon,et al.  Data Association for Multi-Object Tracking via Deep Neural Networks , 2019, Sensors.

[22]  Wei Wu,et al.  Multi-Object Tracking with Multiple Cues and Switcher-Aware Classification , 2019, ArXiv.

[23]  Yue Zhou,et al.  LSTM Multiple Object Tracker Combining Multiple Cues , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[24]  Ming-Hsuan Yang,et al.  Online Multi-Object Tracking with Dual Matching Attention Networks , 2018, ECCV.

[25]  James M. Rehg,et al.  Multi-object Tracking with Neural Gating Using Bilinear LSTM , 2018, ECCV.

[26]  Ihsan Ullah,et al.  Survey on Deep Learning Techniques for Person Re-Identification Task , 2018, ArXiv.

[27]  Long Chen,et al.  Real-Time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[28]  Faouzi Alaya Cheikh,et al.  A Directed Sparse Graphical Model for Multi-target Tracking , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Gabriel Peyré,et al.  Computational Optimal Transport , 2018, Found. Trends Mach. Learn..

[30]  Lu Wang,et al.  Online multiple object tracking via flow and convolutional features , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[31]  Xiao-Feng Li,et al.  Using Stacked Auto-encoder to Get Feature with Continuity and Distinguishability in Multi-object Tracking , 2017, ICIG.

[32]  Bernt Schiele,et al.  Multiple People Tracking by Lifted Multicut and Person Re-identification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Yang Zhang,et al.  Enhancing Detection Model for Multiple Hypothesis Tracking , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[34]  Dietrich Paulus,et al.  Simple online and realtime tracking with a deep association metric , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[35]  Stefano Alletto,et al.  Similarity Mapping with Enhanced Siamese Network for Multi-Object Tracking , 2016, ArXiv.

[36]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Fan Yang,et al.  Exploit All the Layers: Fast and Accurate CNN Object Detector with Scale Dependent Pooling and Cascaded Rejection Classifiers , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Konrad Schindler,et al.  Learning by Tracking: Siamese CNN for Robust Target Association , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[39]  Konrad Schindler,et al.  Online Multi-Target Tracking Using Recurrent Neural Networks , 2016, AAAI.

[40]  Stefan Roth,et al.  MOT16: A Benchmark for Multi-Object Tracking , 2016, ArXiv.

[41]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[44]  Euntai Kim,et al.  Multiple Object Tracking via Feature Pyramid Siamese Networks , 2019, IEEE Access.

[45]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[46]  Cindy Cappelle,et al.  Multi-object tracking with discriminant correlation filter based deep learning tracker , 2019, Integr. Comput. Aided Eng..

[47]  Tian Qi,et al.  Collaborative Deep Reinforcement Learning for Multi-object Tracking , 2018 .

[48]  Kwang-Yong Kim,et al.  Multi-object tracker using kemelized correlation filter based on appearance and motion model , 2017, 2017 19th International Conference on Advanced Communication Technology (ICACT).