The visual keyboard: Real-time feet tracking for the control of musical meta-instruments

This paper proposes a new perceptual interface for the control of computer-based music production. We address the constraints imposed by the use of musical meta-instruments during live performance or rehearsal by tracking feet motion relatively to a visual keyboard. The visual attribute stands for the fact that, unlike its physical counterpart, our keyboard does not involve any force feedback during key-presses. The proposed tracking algorithm is structured on two levels, namely a coarse level for foot regions, and a fine level for foot tips. Tracking works in real-time and handles efficiently feet regions merging/unmerging due to spatial proximity and cast shadows. The output of the tracking is used for the spatiotemporal detection of key-''press'' events.

[1]  M. Betke,et al.  The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[2]  C Baur,et al.  A non-contact mouse for surgeon-computer interaction. , 2004, Technology and health care : official journal of the European Society for Engineering and Medicine.

[3]  R. Behringer.  Conducting digitally stored music by computer vision tracking , 2005, First International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution (AXMEDIS'05).

[4]  Michael Isard,et al.  Contour Tracking by Stochastic Propagation of Conditional Density , 1996, ECCV.

[5]  Nicu Sebe,et al.  Multimodal Human Computer Interaction , 2005 .

[6]  David Merrill,et al.  Head-Tracking for Gestural and Continuous Control of Parameterized Audio Effects , 2003, NIME.

[7]  Shuji Hashimoto,et al.  EyesWeb: Toward Gesture and Affect Recognition in Interactive Dance and Music Systems , 2000, Computer Music Journal.

[8]  Alex Pentland,et al.  Media in performance: Interactive spaces for dance, theater, circus, and museum exhibits , 2000, IBM Syst. J..

[9]  Fumio Miyazaki,et al.  FAce MOUSe: A novel human-machine interface for controlling the position of a laparoscope , 2003, IEEE Trans. Robotics Autom..

[10]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Michael J. Lyons,et al.  Designing, Playing, and Performing with a Vision-based Mouth Interface , 2003, NIME.

[12]  Reinhold Behringer.  Gesture Interaction for Electronic Music Performance , 2007, HCI.

[13]  Todd Winkler,et al.  Making Motion Musical: Gesture Mapping Strategies for Interactive Computer Music , 1995, ICMC.

[14]  Gary R. Bradski,et al.  Motion segmentation and pose recognition with motion history gradients , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[15]  Jan O. Borchers,et al.  Toward a Framework for Interactive Systems to Conduct Digital Audio and Video Streams , 2006, Computer Music Journal.

[16]  Alexandra Branzan Albu,et al.  Computer Vision-Based Interface for the Control of Meta-Instruments , 2007 .

[17]  Michael J. Lyons,et al.  Facing the music: a facial action controlled musical interface , 2001, CHI Extended Abstracts.

[18]  Alex Pentland,et al.  Perceptive Spaces for Performance and Entertainment Untethered Interaction Using Computer Vision and Audition , 1997, Appl. Artif. Intell..

[19]  Terence Sim,et al.  VIM: Vision for Interactive Music , 2007, 2007 IEEE Workshop on Applications of Computer Vision (WACV '07).

[20]  Michael Isard,et al.  A mixed-state condensation tracker with automatic model-switching , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[21]  Gary R. Bradski,et al.  Motion segmentation and pose recognition with motion history gradients , 2002, Machine Vision and Applications.

[22]  W. Andrew Schloss,et al.  Intelligent musical instruments: The future of musical performance or the demise of the performer? , 1993 .

[23]  Nuray Aykin,et al.  Usability and Internationalization. HCI and Culture, Second International Conference on Usability and Internationalization, UI-HCII 2007, Held as Part of HCI International 2007, Beijing, China, July 22-27, 2007, Proceedings, Part I , 2007, HCI.

[24]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[25]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[26]  Ginevra Castellano,et al.  Expressive control of music and visual media by full-body movement , 2007, NIME '07.