Spatio-temporal summarization of dance choreographies

Abstract An important issue in performing dance analysis is the automatic extraction of its choreographic patterns, since these elements provide an abstract representation of the semantics of the dance and encode the overall dance storytelling. However, application of conventional video summarization algorithms on dance sequences cannot appropriately retrieve their choreographic patterns, since a dance is composed of an ordered set of sequential elements which are often repeated in time. Additionally, 3D geometry is lost using color information. For this reason, in this paper we propose a new dance summarization scheme of 3D motion captured data (in the form of skeleton joints coordinates) recorded using the Vicon motion capture system. The proposed key frame extraction method implements a hierarchical scheme that exploits spatio-temporal variations of dance features. Initially, global holistic descriptors are extracted to localize the key choreographic steps of a dance (coarse representation). Then, each segment is further decomposed into finer sub-segments to improve dance representativity (fine representation). The abstraction scheme exploits the concepts of a Sparse Modeling Representative Selection (SMRS) appropriately modified to enable spatio-temporal modelling of the dance sequences through a hierarchical decomposition algorithm. Our approach is evaluated on thirty folkloric dance sequences recorded at the Aristotle University of Thessaloniki under the framework of Terpsichore project representing five different choreographies and on publicly available datasets from Carnegie–Mellon University, which depict performances on theatrical kinesiology. Comparisons with other traditional video summarization methods indicate a clear superiority of the proposed hierarchical spatio-temporal decomposition scheme.