SyncUp

The beauty of synchronized dancing lies in the synchronization of body movements among multiple dancers. While dancers utilize camera recordings for their practice, standard video interfaces do not efficiently support their activities of identifying segments where they are not well synchronized. This thus fails to close a tight loop of an iterative practice process (i.e., capturing a practice, reviewing the video, and practicing again). We present SyncUp, a system that provides multiple interactive visualizations to support the practice of synchronized dancing and liberate users from manual inspection of recorded practice videos. By analyzing videos uploaded by users, SyncUp quantifies two aspects of synchronization in dancing: pose similarity among multiple dancers and temporal alignment of their movements. The system then highlights which body parts and which portions of the dance routine require further practice to achieve better synchronization. The results of our system evaluations show that our pose similarity estimation and temporal alignment predictions were correlated well with human ratings. Participants in our qualitative user evaluation expressed the benefits and its potential use of SyncUp, confirming that it would enable quick iterative practice.

[1]  Aziz Nasridinov,et al.  Pose evaluation for dance learning application using joint position and angular similarity , 2020, UbiComp/ISWC Adjunct.

[2]  Chung Hyuk Park,et al.  Dance with a Robot: Encoder-Decoder Neural Network for Music-Dance Learning , 2020, HRI.

[3]  Congyi Wang,et al.  Music2Dance: DanceNet for Music-Driven Dance Generation , 2020, ACM Trans. Multim. Comput. Commun. Appl..

[4]  Michael J. Black,et al.  VIBE: Video Inference for Human Body Pose and Shape Estimation , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Ting Liu,et al.  View-Invariant Probabilistic Embedding for Human Pose , 2019, ECCV.

[6]  Koji Yatani,et al.  Visualizing Out-of-synchronization in Group Dancing , 2019, UIST.

[7]  Wendy E. Mackay,et al.  Capturing Movement Decomposition to Support Learning and Teaching in Contemporary Dance , 2019, Proc. ACM Hum. Comput. Interact..

[8]  Akrivi Katifori,et al.  Dance Interactive Learning Systems , 2019, ACM Comput. Surv..

[9]  Fan Zhang,et al.  MediaPipe: A Framework for Building Perception Pipelines , 2019, ArXiv.

[10]  Wendy E. Mackay,et al.  Understanding Kinaesthetic Creativity in Dance , 2019, CHI.

[11]  Biao Xie,et al.  Pose-Guided Level Design , 2019, CHI.

[12]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Maneesh Agrawala,et al.  Visual rhythm and beat , 2018, ACM Trans. Graph..

[14]  Antonio Camurri,et al.  A Conceptual Framework for Creating and Analyzing Dance Learning Digital Content , 2018, MOCO.

[15]  Wendy E. Mackay,et al.  How Do Dancers Learn To Dance?: A first-person perspective of dance acquisition by expert contemporary dancers , 2018, MOCO.

[16]  Lei Shi,et al.  Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Wendy E. Mackay,et al.  Knotation: Exploring and Documenting Choreographic Processes , 2018, CHI.

[18]  Francesco Cafaro,et al.  Takes Tutu to Ballet , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[19]  Cewu Lu,et al.  Pose Flow: Efficient Online Pose Tracking , 2018, BMVC.

[20]  Dahua Lin,et al.  Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition , 2018, AAAI.

[21]  Christian Theobalt,et al.  MonoPerfCap , 2017, ACM Trans. Graph..

[22]  Cewu Lu,et al.  RMPE: Regional Multi-person Pose Estimation , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Radoslaw Niewiadomski,et al.  A system to support the learning of movement qualities in dance: a case study on dynamic symmetry , 2016, UbiComp Adjunct.

[24]  Gang Wang,et al.  Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition , 2016, ECCV.

[25]  Eric Brachmann,et al.  Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[26]  Yong Du,et al.  Hierarchical recurrent neural network for skeleton based action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Ling Guan,et al.  An Approach to Ballet Dance Training through MS Kinect and Visualization in a CAVE Virtual Reality Environment , 2015, ACM Trans. Intell. Syst. Technol..

[28]  Falisha J. Karpati,et al.  Dance and the brain: a review , 2015, Annals of the New York Academy of Sciences.

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Aaron C. Courville,et al.  Generative Adversarial Nets , 2014, NIPS.

[31]  Bernt Schiele,et al.  2D Human Pose Estimation: New Benchmark and State of the Art Analysis , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[33]  Andrew Blake,et al.  Efficient Human Pose Estimation from Single Depth Images , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Pavel Zezula,et al.  A Key-Pose Similarity Algorithm for Motion Data Retrieval , 2013, ACIVS.

[36]  Tovi Grossman,et al.  YouMove: enhancing movement training with an augmented reality mirror , 2013, UIST.

[37]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[38]  Emily S. Cross,et al.  Neurocognitive control in dance perception and performance. , 2012, Acta psychologica.

[39]  Yi Yang,et al.  Learning a 3D Human Pose Distance Metric from Geometric Pose Descriptor , 2011, IEEE Transactions on Visualization and Computer Graphics.

[40]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[41]  Taku Komura,et al.  A Virtual Reality Dance Training System Using Motion Capture Technology , 2011, IEEE Transactions on Learning Technologies.

[42]  Dieter Drobny,et al.  Saltate!: a sensor-based system to support dance beginners , 2009, CHI Extended Abstracts.

[43]  Daniel P. W. Ellis,et al.  Beat Tracking by Dynamic Programming , 2007 .

[44]  Meinard Müller,et al.  Efficient content-based retrieval of motion capture data , 2005, ACM Trans. Graph..

[45]  Yoshinori Kuno,et al.  Multimodal presentation method for a dance training system , 2005, CHI Extended Abstracts.

[46]  Eduardo de Campos Valadares,et al.  Dancing to the music , 2000 .

[47]  Alexander J. Smola,et al.  Support Vector Regression Machines , 1996, NIPS.

[48]  David A. Ross,et al.  Learn to Dance with AIST++: Music Conditioned 3D Dance Generation , 2021, ArXiv.

[49]  Alireza Moghaddam,et al.  Coding Issues in Grounded Theory. , 2006 .