Estimating Blink Probability for Highlight Detection in Figure Skating Videos

Highlight detection in sports videos has a broad viewership and huge commercial potential. It is thus imperative to detect highlight scenes more suitably for human interest with high temporal accuracy. Since people instinctively suppress blinks during attention-grabbing events and synchronously generate blinks at attention break points in videos, the instantaneous blink rate can be utilized as a highly accurate temporal indicator of human interest. Therefore, in this study, we propose a novel, automatic highlight detection method based on the blink rate. The method trains a one-dimensional convolution network (1D-CNN) to assess blink rates at each video frame from the spatio-temporal pose features of figure skating videos. Experiments show that the method successfully estimates the blink rate in 94% of the video clips and predicts the temporal change in the blink rate around a jump event with high accuracy. Moreover, the method detects not only the representative athletic action, but also the distinctive artistic expression of figure skating performance as key frames. This suggests that the blink-rate-based supervised learning approach enables high-accuracy highlight detection that more closely matches human sensibility.

[1]  Wei-Ta Chu,et al.  Editing by Viewing: Automatic Home Video Summarization by Viewing Behavior Analysis , 2011, IEEE Transactions on Multimedia.

[2]  Bin Zhao,et al.  HSA-RNN: Hierarchical Structure-Adaptive RNN for Video Summarization , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Nicu Sebe,et al.  Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents , 2010, Multimedia Tools and Applications.

[4]  Michael Lam,et al.  Unsupervised Video Summarization with Adversarial LSTM Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Harry W. Agius,et al.  Analysing user physiological responses for affective video summarisation , 2009, Displays.

[6]  Shih-Fu Chang,et al.  Algorithms and system for segmentation and structure analysis in soccer video , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[7]  Antonio Torralba,et al.  Assessing the Quality of Actions , 2014, ECCV.

[8]  Yue Wang,et al.  Motion-State-Adaptive Video Summarization via Spatiotemporal Analysis , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[9]  Vinod Chandran,et al.  Automatic Identification of Sports Video Highlights using Viewer Interest Features , 2016, ICMR.

[10]  Yueting Zhuang,et al.  Adaptive key frame extraction using unsupervised clustering , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[11]  Luc Van Gool,et al.  Creating Summaries from User Videos , 2014, ECCV.

[12]  Kristen Grauman,et al.  Diverse Sequential Subset Selection for Supervised Video Summarization , 2014, NIPS.

[13]  T. Nakano,et al.  Blink synchronization is an indicator of interest while viewing videos. , 2019, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[14]  Esa Rahtu,et al.  Rethinking the Evaluation of Video Summaries , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Xiangyang Xue,et al.  Learning to Score Figure Skating Sport Videos , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Ke Zhang,et al.  Video Summarization with Long Short-Term Memory , 2016, ECCV.

[17]  T. Nakano,et al.  Blink and you’ll miss it: the role of blinking in the perception of magic tricks , 2016, PeerJ.

[18]  Keiichi Kitajo,et al.  Synchronization of spontaneous eyeblinks while viewing video stories , 2009, Proceedings of the Royal Society B: Biological Sciences.

[19]  Minyi Guo,et al.  Unsupervised Extraction of Video Highlights via Robust Recurrent Auto-Encoders , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  C. Schmid,et al.  Category-Specific Video Summarization , 2014, ECCV.

[22]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[23]  Junqing Yu,et al.  Affection arousal based highlight extraction for soccer video , 2013, Multimedia Tools and Applications.

[24]  Shaohui Mei,et al.  Video summarization via minimum sparse reconstruction , 2015, Pattern Recognit..

[25]  Arnaldo de Albuquerque Araújo,et al.  VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method , 2011, Pattern Recognit. Lett..

[26]  Boon-Lock Yeo,et al.  Analysis And Presentation Of Soccer Highlights From Digital Video , 1995 .

[27]  J. Stern,et al.  The endogenous eyeblink. , 1984, Psychophysiology.

[28]  Qi Tian,et al.  Multilevel video representation with application to keyframe extraction , 2004, 10th International Multimedia Modelling Conference, 2004. Proceedings..

[29]  Jiebo Luo,et al.  Towards Scalable Summarization of Consumer Videos Via Sparse Dictionary Selection , 2012, IEEE Transactions on Multimedia.

[30]  Anoop Gupta,et al.  Automatically extracting highlights for TV Baseball programs , 2000, ACM Multimedia.

[31]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Shih-Fu Chang,et al.  Structure analysis of soccer video with hidden Markov models , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[33]  A. Murat Tekalp,et al.  Generic play-break event detection for summarization and hierarchical sports video analysis , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).