Continuous Realtime Gesture Following and Recognition

We present a HMM based system for real-time gesture analysis. The system outputs continuously parameters relative to the gesture time progression and its likelihood. These parameters are computed by comparing the performed gesture with stored reference gestures. The method relies on a detailed modeling of multidimensional temporal curves. Compared to standard HMM systems, the learning procedure is simplified using prior knowledge allowing the system to use a single example for each class. Several applications have been developed using this system in the context of music education, music and dance performances and interactive installation. Typically, the estimation of the time progression allows for the synchronization of physical gestures to sound files by time stretching/compressing audio buffers or videos.

[1]  Yoram Singer,et al.  The Hierarchical Hidden Markov Model: Analysis and Applications , 1998, Machine Learning.

[2]  Diemo Schwarz,et al.  Ftm - Complex Data Structures for Max , 2005, ICMC.

[3]  Aaron F. Bobick,et al.  Realtime online adaptive gesture recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[4]  F. Pachet,et al.  Eartoy : interactions ludiques par l'audition , 2006 .

[5]  Todd Ingalls,et al.  Real-time Gesture Recognition with Minimal Training Requirements and On-line Learning , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Norbert Schnell,et al.  Wireless sensor interface and gesture-follower for music pedagogy , 2007, NIME '07.

[7]  Aaron F. Bobick,et al.  A State-Based Approach to the Representation and Recognition of Gesture , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Ryo Kurazume,et al.  Early Recognition and Prediction of Gestures , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[9]  Gang Qian,et al.  A Hybrid HMM/DPA Adaptive Gesture Recognition Method , 2005, ISVC.

[10]  Xavier Rodet,et al.  Short-time Viterbi for online HMM decoding: Evaluation on a real-time phone recognition task , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[11]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[12]  Gang Qian,et al.  HMM parameter reduction for practical gesture recognition , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[13]  Norbert Schnell,et al.  MnM: a Max/MSP mapping toolbox , 2005, NIME.

[14]  Remy,et al.  Human Motion Following using Hidden Markov Models and application to dance performance , 2004 .

[15]  Frédéric Bevilacqua,et al.  Effort-based Analysis of Bowing Movements: Evidence of Anticipation Effects , 2008 .

[16]  Nicola Orio,et al.  Robust Polyphonic Midi Score Following with Hidden Markov Models , 2004, ICMC.

[17]  Arshia Cont,et al.  Antescofo: Anticipatory Synchronization and control of Interactive parameters in Computer Music , 2008, ICMC.

[18]  Frédéric Bevilacqua,et al.  Perspectives on Gesture–Sound Relationships Informed from Acoustic Instrument Studies , 2009, Organised Sound.

[19]  Patrick Gallinari,et al.  Online Handwritten Shape Recognition Using Segmental Hidden Markov Models , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.