MV-Sports: A Motion and Vision Sensor Integration-Based Sports Analysis System

Recently, intelligent sports analytics is becoming a hot area in both industry and academia for coaching, practicing tactic and technical analysis. With the growing trend of bringing sports analytics to live broadcasting, sports robots and common playfield, a low cost system that is easy to deploy and performs real-time and accurate sports analytics is very desirable. However, existing systems, such as Hawk-Eye, cannot satisfy these requirements due to various factors. In this paper, we present MV-Sports, a cost-effective system for real-time sports analysis based on motion and vision sensor integration. Taking tennis as a case study, we aim to recognize player shot types and measure ball states. For fine-grained player action recognition, we leverage motion signal for fast action highlighting and propose a long short term memory (LSTM)-based framework to integrate MV data for training and classification. For ball state measurement, we compute the initial ball state via motion sensing and devise an extended kalman filter (EKF)-based approach to combine ball motion physics-based tracking and vision positioning-based tracking to get more accurate ball state. We implement MV-Sports on commercial off-the-shelf (COTS) devices and conduct real-world experiments to evaluate the performance of our system. The results show our approach can achieve accurate player action recognition and ball state measurement with sub-second latency.

[1]  N. Owens,et al.  Hawk-eye tennis system , 2003 .

[2]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[3]  Guang-Zhong Yang,et al.  A Deep Learning Approach to on-Node Sensor Data Analytics for Mobile or Wearable Devices , 2017, IEEE Journal of Biomedical and Health Informatics.

[4]  Hideo Saito,et al.  Reconstruction of 3D Trajectories for Performance Analysis in Table Tennis , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[5]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  De Xu,et al.  Visual Measurement and Prediction of Ball Trajectory for Table Tennis Robot , 2010, IEEE Transactions on Instrumentation and Measurement.

[7]  R. Cross.  The bounce of a ball , 1999 .

[8]  Wu Liu,et al.  Siamese neural network based gait recognition for human identification , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[9]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[10]  Berthold Bäuml,et al.  Realtime perception for catching a flying ball with a mobile humanoid , 2011, 2011 IEEE International Conference on Robotics and Automation.

[11]  Fan Yang,et al.  VM-tracking: Visual-motion sensing integration for real-time human tracking , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[12]  Jia Liu,et al.  Automatic player labeling, tracking and field registration and trajectory mapping in broadcast soccer video , 2011, TIST.

[13]  Wojciech Zaremba,et al.  Learning to Execute , 2014, ArXiv.

[14]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[15]  Fan Yang,et al.  EV-sounding: A visual assisted electronic channel sounding system , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[16]  Carsten Griwodz,et al.  Bagadus: an integrated system for arena sports analytics: a soccer case study , 2013, MMSys.

[17]  Fan Yang,et al.  EV-Matching: Bridging Large Visual Data and Electronic Data for Efficient Surveillance , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  William J. Christmas,et al.  A Tennis Ball Tracking Algorithm for Automatic Annotation of Tennis Match , 2005, BMVC.

[20]  Di Salvo Valter,et al.  Validation of Prozone ®: A new video-based performance analysis system , 2006 .

[21]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[24]  Qi Tian,et al.  Trajectory-Based Ball Detection and Tracking in Broadcast Soccer Video , 2006, IEEE Transactions on Multimedia.

[25]  Pietro Perona,et al.  Fast Feature Pyramids for Object Detection , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Wu Liu,et al.  Deep learning based basketball video analysis for intelligent arena application , 2017, Multimedia Tools and Applications.

[27]  Lei Yang,et al.  Bringing IoT to Sports Analytics , 2017, NSDI.

[28]  Peter F. Sturm,et al.  Pinhole Camera Model , 2014, Computer Vision, A Reference Guide.

[29]  Hua-Tsung Chen,et al.  Physics-based ball tracking and 3D trajectory reconstruction with applications to shooting location estimation in basketball video , 2009, J. Vis. Commun. Image Represent..

[30]  H. Brody,et al.  Physics of the tennis racket , 1979 .