Simple mappings, expressive movement: a qualitative investigation into the end-user mapping design of experienced mid-air musicians

ABSTRACT In a New Interface for Musical Expression (NIME), the design of the relationship between a musician’s actions and the instrument’s sound response is critical in creating instruments that facilitate expressive music performance. A growing body of NIMEs expose this design task to the end performer themselves, leading to the possibility of new insights into NIME mapping design: what can be learned from the mapping design strategies of practicing musicians? This research contributes a qualitative study of four highly experienced users of an end-user mapping instrument to examine their mapping practice. The study reveals that the musicians focus on designing simple, robust mappings that minimize errors, embellishing these control gestures with theatrical ancillary gestures that express metaphors. However, musical expression is hindered by the unintentional triggering of musical events. From these findings, a series of heuristics are presented that can be applied in the future development of NIMEs.

[1]  Ross Kirk,et al.  Mapping Strategies for Musical Performance , 2000 .

[2]  Camille Goudeseune,et al.  Interpolated mappings for musical instruments , 2002, Organised Sound.

[3]  Christopher Dobrian,et al.  The 'E' in NIME: Musical Expression with New Computer Interfaces , 2006, NIME.

[4]  Jacob Harrison,et al.  An adapted bass guitar for one-handed playing , 2017, NIME.

[5]  Phoebe Sengers,et al.  The epistemology & evaluation of experience-focused hci , 2009 .

[6]  R. Gibbs The Cambridge Handbook of Metaphor and Thought , 2008 .

[7]  Paul Dourish,et al.  Where the action is , 2001 .

[8]  Stefania Serafin,et al.  Longitudinal Evaluation of the Integration of Digital Musical Instruments into Existing Compositional Work Processes , 2012 .

[9]  Perry R. Cook,et al.  Toward Understanding Human-Computer Interaction In Composing The Instrument , 2010, ICMC.

[10]  Ali Momeni,et al.  Dynamic Independent Mapping Layers for Concurrent Control of Audio and Video Synthesis , 2006, Computer Music Journal.

[11]  Doug Van Nort,et al.  Conducting the in-between: improvisation and intersubjective engagement in soundpainted electro-acoustic ensemble performance , 2018, Digit. Creativity.

[12]  Tom Mitchell,et al.  A user experience review of music interaction evaluations , 2017, NIME.

[13]  Liam J. Bannon,et al.  The turn to practice in HCI: towards a research agenda , 2014, CHI.

[14]  Johan Kildal,et al.  Non-intrusive Counter-actions: Maintaining Progressively Engaging Interactions for Music Performance , 2016, NIME.

[15]  R. Elliott,et al.  Evolving guidelines for publication of qualitative research studies in psychology and related fields. , 1999, The British journal of clinical psychology.

[16]  Geraldine Fitzpatrick,et al.  HCI Methodology For Evaluating Musical Controllers: A Case Study , 2008, NIME.

[17]  Alan F. Blackwell,et al.  The reification of metaphor as a design tool , 2006, TCHI.

[18]  Diana Santos,et al.  The E , 2003 .

[19]  Michael Gurevich,et al.  Expression and its discontents: toward an ecology of musical creation , 2007, NIME '07.

[20]  Alan Cienki,et al.  Metaphor, gesture, and thought , 2008 .

[21]  Caroline Hummels,et al.  Designing for Embodied Being-in-the-World: Two Cases, Seven Principles and One Framework , 2017, TEI.

[22]  Mary Mainsbridge Gesture-controlled musical performance: from movement awareness to mastery , 2018 .

[23]  David Collins,et al.  Real-time tracking of the creative music composition process , 2007, Digit. Creativity.

[24]  Jules Françoise,et al.  Shaping and exploring interactive motion-sound mappings using online clustering techniques , 2017, NIME.

[25]  Nick Bryan-Kinns,et al.  Skip the Pre-Concert Demo: How Technical Familiarity and Musical Style Affect Audience Response , 2016, NIME.

[26]  Steve Larson,et al.  "Something in the Way She Moves"-Metaphors of Musical Motion , 2003 .

[27]  Tom Mitchell,et al.  Understanding User-Defined Mapping Design in Mid-Air Musical Performance , 2018, MOCO.

[28]  S. M. Hansen Where the action is. , 1985, Journal (National Association for Hospital Development (U.S.)).

[29]  Marcelo M. Wanderley,et al.  The Importance of Parameter Mapping in Electronic Instrument Design , 2002, NIME.

[30]  Felipe Otondo Using mobile sound to explore spatial relationships between dance and music performance , 2018, Digit. Creativity.

[31]  鐘期 坂本,et al.  Tonal Pitch Space を用いた楽曲の和声解析 , 2009 .

[32]  Gerhard Fischer,et al.  End-user modifiability in design environments , 1990, CHI '90.

[33]  Alex Hofmann,et al.  From interface design to the software instrument - mapping as an approach to fx-instrument building , 2017, NIME.

[34]  Perry R. Cook,et al.  A Meta-Instrument for Interactive, On-the-Fly Machine Learning , 2009, NIME.

[35]  Tony Stockman,et al.  Rich gesture, reduced control: the influence of constrained mappings on performance technique , 2017, MOCO.

[36]  James L. McClelland,et al.  Phenomenology of perception. , 1978, Science.

[37]  Garth Paine,et al.  The Thummer Mapping Project (ThuMP) , 2007, NIME '07.

[38]  Johannes Birringer,et al.  Wearable performance , 2009, Digit. Creativity.

[39]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[40]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[41]  Richard Taruskin,et al.  Text and Act: Essays on Music and Performance , 1996 .

[42]  Norbert Schnell,et al.  Mapping Through Listening , 2014, Computer Music Journal.

[43]  Kristofer Klipfel MIDI Motion: Interactive Music Composition Gloves , 2017, Tangible and Embedded Interaction.

[44]  Jules Françoise Gesture--sound mapping by demonstration in interactive music systems , 2013, MM '13.

[45]  Marcelo M. Wanderley,et al.  Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance , 1997 .

[46]  Susanne Bødker,et al.  Third-wave HCI, 10 years later---participation and sharing , 2015, Interactions.

[47]  Arnie Cox Music and Embodied Cognition , 2016 .

[48]  Thomas Mitchell,et al.  SoundGrasp: A Gestural Interface for the Performance of Live Music , 2011, NIME.

[49]  Caroline Hummels,et al.  The Blue Studio: Designing an Interactive Environment for Embodied Multi-Stakeholder Ideation Processes , 2017, Tangible and Embedded Interaction.

[50]  P. Janata,et al.  Embodied music cognition and mediation technology , 2009 .

[51]  Paul Mulholland,et al.  What Can the Language of Musicians Tell Us about Music Interaction Design? , 2010, Computer Music Journal.

[52]  Andrew P. McPherson,et al.  Action-Sound Latency: Are Our Tools Fast Enough? , 2016, NIME.

[53]  Matthew Wright,et al.  Problems and Prospects for Intimate Musical Control of Computers , 2002, Computer Music Journal.

[54]  Thomas Mitchell,et al.  Musical Interaction with Hand Posture and Orientation: A Toolbox of Gestural Control Mechanisms , 2012, NIME.

[55]  Jules Françoise Motion-Sound Mapping by Demonstration , 2015 .

[56]  F. R. Wilson The Hand: How Its Use Shapes the Brain, Language, and Human Culture , 1998 .

[57]  Sidney Fels,et al.  Mapping transparency through metaphor: towards more expressive musical instruments , 2002, Organised Sound.

[58]  Andrew P. McPherson,et al.  The space between the notes: adding expressive pitch control to the piano keyboard , 2013, CHI.

[59]  Marcelo M. Wanderley,et al.  From controller to sound: Tools for Collaborative Development of Digital Musical Instruments , 2007, ICMC.

[60]  Eduardo Reck Miranda,et al.  A knowledge-based, data-driven method for action-sound mapping , 2017, NIME.

[61]  J. Pallasmaa Embodied and Existential Wisdom in Architecture , 2017 .

[62]  G. Lakoff,et al.  Metaphors We Live By , 1980 .

[63]  Antonella De Angeli,et al.  Musical Interface Design: An Experience-oriented Framework , 2014, NIME.

[64]  Andrew Johnston,et al.  Interfaces for musical expression based on simulated physical models , 2009 .

[65]  Kevin G. Munhall,et al.  Something in the way she moves , 2004, Trends in Cognitive Sciences.

[66]  Youngmoo E. Kim,et al.  The Problem of the Second Performer: Building a Community Around an Augmented Piano , 2012, Computer Music Journal.

[67]  Nicola Orio,et al.  Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI , 2001, Computer Music Journal.

[68]  Chris Kiefer,et al.  Self-resonating feedback cello: interfacing gestural and generative processes in improvised performance , 2017, NIME.