Enhancing spatial perception through sound: mapping human movements into MIDI

Gestural expressiveness plays a fundamental role in the interaction with people, environments, animals, things, and so on. Thus, several emerging application domains would exploit the interpretation of movements to support their critical designing processes. To this end, new forms to express the people’s perceptions could help their interpretation, like in the case of music. In this paper, we investigate the user’s perception associated with the interpretation of sounds by highlighting how sounds can be exploited for helping users in adapting to a specific environment. We present a novel algorithm for mapping human movements into MIDI music. The algorithm has been implemented in a system that integrates a module for real-time tracking of movements through a sample based synthesizer using different types of filters to modulate frequencies. The system has been evaluated through a user study, in which several users have participated in a room experience, yielding significant results about their perceptions with respect to the environment they were immersed.

[1]  Keith M. Welker,et al.  Music as an emotion regulation strategy: An examination of genres of music and their roles in emotion regulation , 2019 .

[2]  Rykhard Bohush,et al.  Robust Person Tracking Algorithm Based on Convolutional Neural Network for Indoor Video Surveillance Systems , 2019, Communications in Computer and Information Science.

[3]  Francis Rumsey.  MIDI systems and control , 1990 .

[4]  Giuseppe Polese,et al.  EDCAR: A knowledge representation framework to enhance automatic video surveillance , 2019, Expert Syst. Appl..

[5]  Giuseppe Polese,et al.  Iconic language design for people with significant speech and multiple impairments , 1994, Assistive Technology and Artificial Intelligence.

[6]  Wei Niu,et al.  Human activity detection and recognition for video surveillance , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[7]  Yaser Sheikh,et al.  Efficient Online Multi-Person 2D Pose Tracking With Recurrent Spatio-Temporal Affinity Fields , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Horst Bischof,et al.  On-line Boosting and Vision , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[9]  Bruce A. Draper,et al.  Visual object tracking using adaptive correlation filters , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[10]  Hermann von Helmholtz,et al.  On the Sensations of Tone , 1954 .

[11]  Teresa Marrin Nakra,et al.  The "Conductor's Jacket": A Device for Recording Expressive Musical Gestures , 1998, ICMC.

[12]  Yu Chen,et al.  Kerman: A Hybrid Lightweight Tracking Algorithm to Enable Smart Surveillance as an Edge Service , 2019, 2019 16th IEEE Annual Consumer Communications & Networking Conference (CCNC).

[13]  F. Richard Moore,et al.  The Dysfunctions of MIDI , 1988, ICMC.

[14]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Ming-Hsuan Yang,et al.  Robust Object Tracking with Online Multiple Instance Learning , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[17]  Ahmad Jalal,et al.  Multi-Person Tracking in Smart Surveillance System for Crowd Counting and Normal/Abnormal Events Detection , 2019, 2019 International Conference on Applied and Engineering Mathematics (ICAEM).

[18]  Shuangquan Wang,et al.  Ultigesture: A wristband-based platform for continuous gesture control in healthcare , 2019, Smart Health.

[19]  Gennaro Costagliola,et al.  Monitoring Online Tests through Data Visualization , 2009, IEEE Transactions on Knowledge and Data Engineering.

[20]  Jong-Kwan Song,et al.  Gender Classification Based on Binary Haar Cascade , 2014 .

[21]  Emanuele Frontoni,et al.  Preterm Infants’ Pose Estimation With Spatio-Temporal Features , 2019, IEEE Transactions on Biomedical Engineering.

[22]  Insook Choi,et al.  Interfacing sound synthesis to movement for exploring high-dimensional systems in a virtual environment , 1995, 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century.

[23]  Gennaro Costagliola,et al.  Do you like my outfit?: Cromnia, a mobile assistant for blind users , 2018, Goodtechs '18.

[24]  Meng-Luen Wu,et al.  Real-Time Musical Conducting Gesture Recognition Based on a Dynamic Time Warping Classifier Using a Single-Depth Camera , 2019, Applied Sciences.

[25]  David Miles Huber.  The MIDI Manual: A Practical Guide to MIDI in the Project Studio , 1998 .

[26]  Nikos Pelekis,et al.  The DAEDALUS framework: progressive querying and mining of movement data , 2008, GIS '08.

[27]  Adil Alpkocak,et al.  Melody extraction on MIDI music files , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).

[28]  Alberto Del Bimbo,et al.  Real-time hand status recognition from RGB-D imagery , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[29]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[30]  Rui Caseiro,et al.  High-Speed Tracking with Kernelized Correlation Filters , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.