Multi-camera multi-player tracking with deep player identification in sports video

Abstract Identity switches caused by inter-object interactions remain a critical problem for multi-player tracking in real-world sports video analysis. Existing approaches utilizing the appearance model is difficult to associate detections and preserve identities due to the similar appearance of players in the same team. Instead of the appearance model, we propose a distinguishable deep representation for player identity in this paper. A robust multi-player tracker incorporating with deep player identification is further developed to produce identity-coherent trajectories. The framework consists of three parts: (1) the core component, a Deep Player Identification (DeepPlayer) model that provides an adequate discriminative feature through the coarse-to-fine jersey number recognition and the pose-guided partial feature embedding; (2) an Individual Probability Occupancy Map (IPOM) model for players 3D localization with ID; and (3) a K-Shortest Path with ID (KSP-ID) model that links nodes in the flow graph by a proposed player ID correlation coefficient. With the distinguishable identity, the performance of tracking is improved. Experiment results illustrate that our framework handles the identity switches effectively, and outperforms state-of-the-art trackers on the sports video benchmarks.

[1]  Tae-Hyun Oh,et al.  Part-Based Player Identification Using Deep Convolutional Representation and Multi-scale Pooling , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Min Xu,et al.  Synthetic IR Image Refinement Using Adversarial Learning With Bidirectional Mappings , 2019, IEEE Access.

[3]  Xinbo Gao,et al.  Tactic analysis based on real-world ball trajectory in soccer video , 2012, Pattern Recognit..

[4]  James J. Little,et al.  Learning to Track and Identify Players from Broadcast Sports Videos , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Tomislav Pribanic,et al.  3D registration based on the direction sensor measurements , 2019, Pattern Recognit..

[6]  Ruigang Yang,et al.  Semi-Supervised Video Object Segmentation with Super-Trajectories , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Ruigang Yang,et al.  Inferring Salient Objects from Human Fixations , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Karsten Müller,et al.  Soccer player recognition using spatial constellation features and jersey number recognition , 2017, Comput. Vis. Image Underst..

[9]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[10]  Bingbing Ni,et al.  Deep Regression Tracking with Shrinkage Loss , 2018, ECCV.

[11]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Pascal Fua,et al.  Tracking multiple people under global appearance constraints , 2011, 2011 International Conference on Computer Vision.

[13]  Konrad Schindler,et al.  Globally Optimal Multi-target Tracking on a Hexagonal Lattice , 2010, ECCV.

[14]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Changhu Wang,et al.  Jersey Number Recognition with Semi-Supervised Spatial Transformer Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  Masaki Hayashi,et al.  Multiple players tracking and identification using group detection and player number recognition in sports video , 2013, IECON 2013 - 39th Annual Conference of the IEEE Industrial Electronics Society.

[19]  Alberto Del Bimbo,et al.  Matching Faces with Textual Cues in Soccer Videos , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[20]  Fatih Murat Porikli,et al.  Saliency-aware geodesic video object segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Taiki Sekii.  Robust, Real-Time 3D Tracking of Multiple Objects with Similar Appearances , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Ling Shao,et al.  See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Xiubao Sui,et al.  Adaptive pedestrian tracking via patch-based features and spatial-temporal similarity measurement , 2016, Pattern Recognit..

[24]  Mubarak Shah,et al.  Tracking Multiple Occluding People by Localizing on Multiple Scene Planes , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Thomas Mauthner,et al.  Robust Real-Time Tracking of Multiple Objects by Volumetric Mass Densities , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Sanyuan Zhao,et al.  Multiple people tracking with articulation detection and stitching strategy , 2020, Neurocomputing.

[27]  Bernt Schiele,et al.  Multiple People Tracking by Lifted Multicut and Person Re-identification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Francesco Solera,et al.  Performance Measures and a Data Set for Multi-target, Multi-camera Tracking , 2016, ECCV Workshops.

[29]  Min Xu,et al.  3D Multiview Basketball Players Detection and Localization Based on Probabilistic Occupancy , 2018, 2018 Digital Image Computing: Techniques and Applications (DICTA).

[30]  Tiziana D'Orazio,et al.  A review of vision-based systems for soccer video analysis , 2010, Pattern Recognit..

[31]  Karsten Müller,et al.  Soccer Jersey Number Recognition Using Convolutional Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[32]  Pascal Fua,et al.  Multi-Commodity Network Flow for Tracking Multiple People , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Pascal Fua,et al.  Multicamera People Tracking with a Probabilistic Occupancy Map , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[35]  Pascal Fua,et al.  Deep Occlusion Reasoning for Multi-camera Multi-target Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Xiaofeng Xu,et al.  Facial Component-Landmark Detection With Weakly-Supervised LR-CNN , 2019, IEEE Access.

[37]  Jianbing Shen,et al.  Fast Online Tracking With Detection Refinement , 2018, IEEE Transactions on Intelligent Transportation Systems.

[38]  Yanxi Liu,et al.  Tracking Sports Players with Context-Conditioned Motion Models , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Laura Leal-Taixé,et al.  Tracking Without Bells and Whistles , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[40]  Yu Yang,et al.  Research on simulated infrared image utility evaluation using deep representation , 2018 .

[41]  Ákos Utasi,et al.  A 3-D marked point process model for multi-view people detection , 2011, CVPR 2011.

[42]  Pascal Fua,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Multiple Object Tracking Using K-shortest Paths Optimization , 2022 .

[43]  Pierre-Marc Jodoin,et al.  High-speed transition patterns for video projection, 3D reconstruction, and copyright protection , 2015, Pattern Recognit..

[44]  Robert T. Collins,et al.  Hybrid Stochastic / Deterministic Optimization for Tracking Sports Players and Pedestrians , 2014, ECCV.

[45]  Huchuan Lu,et al.  Pose-Invariant Embedding for Deep Person Re-Identification , 2017, IEEE Transactions on Image Processing.

[46]  Yonghong Tian,et al.  Robust multiple cameras pedestrian detection with multi-view Bayesian network , 2015, Pattern Recognit..

[47]  Hitoshi Sato,et al.  Multiple sports player tracking system based on graph optimization using low-cost cameras , 2018, 2018 IEEE International Conference on Consumer Electronics (ICCE).

[48]  Yael Moses,et al.  Homography based multiple camera detection and tracking of people in a dense crowd , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[49]  Wenguan Wang,et al.  Super-Trajectory for Video Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[50]  Wen Gao,et al.  Jersey number detection in sports video for athlete identification , 2005, Visual Communications and Image Processing.