Graph Convolutional Dictionary Selection With L₂,ₚ Norm for Video Summarization

Video Summarization (VS) has become one of the most effective solutions for quickly understanding a large volume of video data. Dictionary selection with self representation and sparse regularization has demonstrated its promise for VS by formulating the VS problem as a sparse selection task on video frames. However, existing dictionary selection models are generally designed only for data reconstruction, which results in the neglect of the inherent structured information among video frames. In addition, the sparsity commonly constrained by <inline-formula> <tex-math notation="LaTeX">$L_{2,1}$ </tex-math></inline-formula> norm is not strong enough, which causes the redundancy of keyframes, i.e., similar keyframes are selected. Therefore, to address these two issues, in this paper we propose a general framework called graph convolutional dictionary selection with <inline-formula> <tex-math notation="LaTeX">$L_{2,p}$ </tex-math></inline-formula> (<inline-formula> <tex-math notation="LaTeX">$0< p\leq 1$ </tex-math></inline-formula>) norm (GCDS<inline-formula> <tex-math notation="LaTeX">$_{2,p}$ </tex-math></inline-formula>) for both keyframe selection and skimming based summarization. Firstly, we incorporate graph embedding into dictionary selection to generate the graph embedding dictionary, which can take the structured information depicted in videos into account. Secondly, we propose to use <inline-formula> <tex-math notation="LaTeX">$L_{2,p}$ </tex-math></inline-formula> (<inline-formula> <tex-math notation="LaTeX">$0< p\leq 1$ </tex-math></inline-formula>) norm constrained row sparsity, in which <inline-formula> <tex-math notation="LaTeX">$p$ </tex-math></inline-formula> can be flexibly set for two forms of video summarization. For keyframe selection, <inline-formula> <tex-math notation="LaTeX">$0< p< 1$ </tex-math></inline-formula> can be utilized to select diverse and representative keyframes; and for skimming, <inline-formula> <tex-math notation="LaTeX">$p=1$ </tex-math></inline-formula> can be utilized to select key shots. In addition, an efficient iterative algorithm is devised to optimize the proposed model, and the convergence is theoretically proved. Experimental results including both keyframe selection and skimming based summarization on four benchmark datasets demonstrate the effectiveness and superiority of the proposed method.