HappyFeet: Recognizing and Assessing Dance on the Floor

The widespread availability of Internet-of-Thing (IoT) devices, wearable sensors and smart watches have been promoting innovative activity recognition applications in our everyday lives. Recognizing dance steps with fine granularity using wearables is one of those exciting applications. In a typical dance classroom scenario where the instructors are frequently outnumbered by the students, accelerometer sensors can be utilized to automatically compare the performance of the dancers and provide informative feedback to all the stakeholders, for example, the instructors and the learners. However, owing to the complexity of the movement kinematics of human body, building a sufficiently accurate and reliable system can be a daunting task. Utilization of multiple sensors can help improve the reliability, however most wearable sensors do not boast sufficient resolution for such tasks and often suffer from various data sampling, device heterogeneity and instability issues. To address these challenges, we introduce \emph{HappyFeet}, a convolutional neural network based deep, self-evolving feature learning model that accurately recognizes the micro steps of various dance activities. We show that our model consistently outperforms feature engineering based shallow learning approaches by a margin (approximately 7%) accuracy on data collected from dance routines (Indian classical) performed by a professional dancer. We also posit a Body Sensor Network model and discuss the underpinning challenges and possible solutions associated with multiple sensors' signal variations.

[1]  Diane J. Cook,et al.  Activity Learning: Discovering, Recognizing, and Predicting Human Behavior from Sensor Data , 2015 .

[2]  Augusto Dias Pereira dos Santos.  Smart Technology for Supporting Dance Education , 2017, UMAP.

[3]  Yiannis Kompatsiaris,et al.  Classifying Salsa dance steps from skeletal poses , 2016, 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI).

[4]  Stephanos Camarinopoulos,et al.  A Study on the Use of Kinect Sensor in Traditional Folk Dances Recognition via Posture Analysis , 2017, PETRA.

[5]  Mikkel Baun Kjærgaard,et al.  Smart Devices are Different: Assessing and MitigatingMobile Sensing Heterogeneities for Activity Recognition , 2015, SenSys.

[6]  Joëlle Tilmanne,et al.  Dance performance evaluation using hidden Markov models , 2016, Comput. Animat. Virtual Worlds.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Uma Shanker Tiwary,et al.  Classification of Indian Classical Dance Forms , 2016, IHCI.

[9]  Hao Wu,et al.  Using automatic generation of Labanotation to protect folk dance , 2017, J. Electronic Imaging.

[10]  Sriparna Saha,et al.  Human skeleton matching for e-learning of dance using a probabilistic neural network , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[11]  Karim Tabia,et al.  Dance Training Tool Using Kinect-Based Skeleton Tracking and Evaluating Dancer's Performance , 2017, IEA/AIE.

[12]  DongSheng Cai,et al.  Nonlinear dance motion analysis and motion editing using Hilbert-Huang transform , 2017, CGI.

[13]  Dohyung Kim,et al.  Classification of K-Pop Dance Movements Based on Skeleton Information Obtained by a Kinect Sensor , 2017, Sensors.

[14]  Young Soo Suh,et al.  Dance training system using foot mounted sensors , 2015, 2015 IEEE/SICE International Symposium on System Integration (SII).

[15]  Hedda Lausberg,et al.  Methods in Gesture Research: , 2009 .

[16]  David V. Thiel,et al.  Accelerometer based performance assessment of basic routines in classical ballet , 2014 .

[17]  Chris Donahue,et al.  Dance Dance Convolution , 2017, ICML.

[18]  Mourad Oussalah,et al.  Analysis of the Movement Variability in Dance Activities Using Wearable Sensors , 2017 .

[19]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[20]  Kavita A. Gavand,et al.  Physical Activity in Youth Dance Classes , 2015, Pediatrics.

[21]  Yejin Kim,et al.  Dance motion capture and composition using multiple RGB and depth sensors , 2017, Int. J. Distributed Sens. Networks.

[22]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[23]  K. Fox,et al.  Dance, Dance Evolution: Accelerometer Sensor Networks as Input to Video Games , 2007, 2007 IEEE International Workshop on Haptic, Audio and Visual Environments and Games.