The Body as Sound: Unpacking Vocal Embodiment through Auditory Biofeedback

Multi-sensory experiences underpin embodiment, whether with the body itself or technological extensions of it. Vocalists experience intensely personal embodiment, as vocalisation has few outwardly visible effects and kinaesthetic sensations occur largely within the body, rather than through external touch. We explored this embodiment using a probe which sonified laryngeal muscular movements and provided novel auditory feedback to two vocalists over a month-long period. Somatic and micro-phenomenological approaches revealed that the vocalists understand their physiology through its sound, rather than awareness of the muscular actions themselves. The feedback shaped the vocalists’ perceptions of their practice and revealed a desire for reassurance about exploration of one’s body when the body-as-sound understanding was disrupted. Vocalists experienced uncertainty and doubt without affirmation of perceived correctness. This research also suggests that technology is viewed as infallible and highlights expectations that exist about its ability to dictate success, even when we desire or intend to explore.

[1]  William W. Gaver,et al.  Emergence as a Feature of Practice-based Design Research , 2022, Conference on Designing Interactive Systems.

[2]  Andrew P. McPherson,et al.  Super Size Me: Interface Size, Identity and Embodiment in Digital Musical Instrument Design , 2022, CHI.

[3]  Laura Devendorf,et al.  Sketching Across the Senses: Exploring Sensory Translation as a Generative Practice for Designing Data Representations , 2022, CHI Extended Abstracts.

[4]  Andrew P. McPherson,et al.  Singing Knit: Soft Knit Biosensing for Augmenting Vocal Performances , 2022, AHs.

[5]  Andrew P. McPherson,et al.  Exploring Experiences with New Musical Instruments through Micro-phenomenology , 2022, NIME.

[6]  Sarah E. Fox,et al.  Cracks in the Success Narrative: Rethinking Failure in Design Research through a Retrospective Trioethnography , 2021, ACM Trans. Comput. Hum. Interact..

[7]  Kristina Höök,et al.  Unpacking Non-Dualistic Design: The Soma Design Case , 2021, ACM Trans. Comput. Hum. Interact..

[8]  Brian Keegan,et al.  Examining Narrative Sonification: Using First-Person Retrospection Methods to Translate Radio Production to Interaction Design , 2021, ACM Trans. Comput. Hum. Interact..

[9]  Paweł W. Woźniak,et al.  Facilitating Bodily Insights Using Electromyography-Based Biofeedback during Physical Activity , 2021, MobileHCI.

[10]  Ozgun Kilic Afsar,et al.  SymbioSinging: Robotically transposing singing experience across singing and non-singing bodies , 2021, Creativity & Cognition.

[11]  Albrecht Schmidt,et al.  EMBody: A Data-Centric Toolkit for EMG-Based Interface Prototyping and Experimentation , 2021, Proc. ACM Hum. Comput. Interact..

[12]  Albrecht Schmidt,et al.  EMPiano: Electromyographic Pitch Control on the Piano Keyboard , 2021, CHI Extended Abstracts.

[13]  Sarah Homewood,et al.  Tracing Conceptions of the Body in HCI: From User to More-Than-Human , 2021, CHI.

[14]  Corina Sas,et al.  Interoceptive Interaction: An Embodied Metaphor Inspired Approach to Designing for Meditation , 2021, CHI.

[15]  Vasiliki Tsaknaki,et al.  “Feeling the Sensor Feeling you”: A Soma Design Exploration on Sensing Non-habitual Breathing , 2021, CHI.

[16]  Vasiliki Tsaknaki,et al.  The Body Electric: A NIME designed through and with the somatic experience of singing , 2021, NIME.

[17]  S. Waters The entanglements which make instruments musical: Rediscovering sociality , 2021, Journal of New Music Research.

[18]  Katta Spiel The Bodies of TEI – Investigating Norms and Assumptions in the Design of Embodied Interaction , 2021, TEI.

[19]  Andrew P. McPherson,et al.  Surface Electromyography for Sensing Performance Intention and Musical Imagery in Vocalists , 2021, TEI.

[20]  R. T. Smith Story , 2006, Storytelling Exhibitions.

[21]  Albrecht Schmidt,et al.  Hit the Thumb Jack! Using Electromyography to Augment the Piano Keyboard , 2020, Conference on Designing Interactive Systems.

[22]  Mirjana Prpa,et al.  Inhaling and Exhaling: How Technologies Can Perceptually Extend our Breath Awareness , 2020, CHI.

[23]  Mirjana Prpa,et al.  Articulating Experience: Reflections from Experts Applying Micro-Phenomenology to Design Research in HCI , 2020, CHI.

[24]  Wendy Ju,et al.  Next Steps for Human-Computer Integration , 2020, CHI.

[25]  Marianne Clark,et al.  Towards Diffractive Ways of Knowing Women’s Moving Bodies: A Baradian Experiment With the Fitbit–Motherhood Entanglement , 2020 .

[26]  Mike Y. Chen,et al.  MuscleSense: Exploring Weight Sensing using Wearable Surface Electromyography (sEMG) , 2020, TEI.

[27]  Ron Wakkary,et al.  Digital crafts-machine-ship , 2019, Interactions.

[28]  Christopher Frauenberger,et al.  Entanglement HCI The Next Wave? , 2019, ACM Trans. Comput. Hum. Interact..

[29]  Andrew P. McPherson,et al.  Surface Electromyography for Direct Vocal Control , 2020, NIME.

[30]  Andrew P. McPherson,et al.  Soma Design for NIME , 2020 .

[31]  Alexander Refsum Jensenius,et al.  RAW: Exploring Control Structures for Muscle-based Interaction in Collective Improvisation , 2020 .

[32]  M. Cecília C. Baranauskas,et al.  The Interface between Interactive Art and Human-Computer Interaction: Exploring Dialogue Genres and Evaluative Practices , 2019 .

[33]  Kristina Höök,et al.  Designing with the Body: Somaesthetic Interaction Design , 2018, CHIRA.

[34]  S. Dahl,et al.  Communicating Gait Performance Through Musical Energy: Towards an Intuitive Biofeedback System for Neurororehabilitation , 2019 .

[35]  C. Valenzuela-Moguillansky,et al.  An Analysis Procedure for the Micro-Phenomenological Interview , 2019 .

[36]  C. Petitmengin,et al.  Discovering the structures of lived experience , 2018, Phenomenology and the Cognitive Sciences.

[37]  Thecla Schiphorst,et al.  The somatic turn in human-computer interaction , 2018, Interactions.

[38]  B. Riecke,et al.  Immersive Interactive Technologies for Positive Change: A Scoping Review and Design Considerations , 2018, Front. Psychol..

[39]  Albrecht Schmidt,et al.  EMGuitar: Assisting Guitar Playing with Electromyography , 2018, Conference on Designing Interactive Systems.

[40]  Alexander Refsum Jensenius,et al.  Composing an Ensemble Standstill Work for Myo and Bela , 2018, NIME.

[41]  Kristina Höök,et al.  Embracing First-Person Perspectives in Soma-Based Design , 2018, Informatics.

[42]  Deborah E. White,et al.  Thematic Analysis , 2017 .

[43]  Antti Pirhonen,et al.  Who Controls Who? Embodied Control Within Human-Technology Choreographies† , 2017, Interact. Comput..

[44]  Andrew P. McPherson,et al.  Bela: An embedded platform for low-latency feedback control of sound , 2017 .

[45]  Miguel Ortiz,et al.  Gestural Musical Performance with Physiological Sensors, Focusing on the Electromyogram , 2017 .

[46]  Lian Loke,et al.  Felt-sensing archetypes: analysing patterns of accessing tacit meaning in design , 2016, OZCHI.

[47]  Federico Casalegno,et al.  EMG-based biofeedback tool for augmenting manual fabrication and improved exchange of empirical knowledge , 2015, Interacción.

[48]  Claudia Núñez-Pacheco,et al.  Expanding our perceptual world through technology: a subjective bodily perspective , 2015, UbiComp/ISWC Adjunct.

[49]  Jessica O'Bryan “We ARE our instrument!”: Forming a singer identity , 2015 .

[50]  Atau Tanaka,et al.  Intention, Effort, and Restraint: The EMG in Musical Performance , 2015, Leonardo.

[51]  Andrew P. McPherson,et al.  An Environment for Submillisecond-Latency Audio and Sensor Processing on BeagleBone Black , 2015 .

[52]  P. Verbeek COVER STORYBeyond interaction , 2015 .

[53]  Toni Robertson,et al.  Moving and making strange: An embodied approach to movement-based interaction design , 2013, TCHI.

[54]  Dag Svanæs,et al.  Interaction design for and with the lived body: Some implications of merleau-ponty's phenomenology , 2013, TCHI.

[55]  Atau Tanaka,et al.  Muscular Interactions. Combining EMG and MMG sensing for musical practice , 2013, NIME.

[56]  E. Bates The social life of musical instruments , 2012 .

[57]  Dorothy Miell,et al.  Musical Imaginations: Multidisciplinary perspectives on creativity, performance and perception , 2011 .

[58]  Thecla Schiphorst,et al.  Self-evidence: applying somatic connoisseurship to experience design , 2011, CHI Extended Abstracts.

[59]  Jennifer Aileen Jestley Metaphorical and non-metaphorical imagery use in vocal pedagogy : an investigation of underlying cognitive organisational constructs , 2011 .

[60]  Sarah Nicolls,et al.  Seeking Out the Spaces Between: Using Improvisation in Collaborative Composition with Interactive Technology , 2010, Leonardo Music Journal.

[61]  Jakub Matyja,et al.  Embodied Music Cognition , 2010 .

[62]  Kenji Suzuki,et al.  BioTones: A wearable device for EMG auditory biofeedback , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[63]  Kristina Höök,et al.  Transferring qualities from horseback riding to design , 2010, NordiCHI.

[64]  Inger Ekman,et al.  Using vocal sketching for designing sonic interactions , 2010, Conference on Designing Interactive Systems.

[65]  Shaowen Bardzell,et al.  Feminist HCI: taking stock and outlining an agenda for design , 2010, CHI.

[66]  Anthony F. Beavers The Phenomenological Mind: An Introduction to Philosophy of Mind and Cognitive Science , 2009 .

[67]  P. Janata,et al.  Embodied music cognition and mediation technology , 2009 .

[68]  Marc Leman,et al.  The musical instrument as a natural extension of the musician , 2009 .

[69]  P. Vermersch,et al.  Describing the Practice of Introspection , 2009 .

[70]  Greg Corness,et al.  What the body knows: Exploring the benefits of embodied metaphors in hybrid physical digital environments , 2009, Interact. Comput..

[71]  P. Soucacos,et al.  Functional neuroanatomy of proprioception. , 2008, Journal of surgical orthopaedic advances.

[72]  Richard Shusterman,et al.  Body Consciousness: A Philosophy of Mindfulness and Somaesthetics , 2008, The Journal of Speculative Philosophy.

[73]  S. Gallagher,et al.  The Phenomenological Mind: An Introduction to Philosophy of Mind and Cognitive Science , 2007 .

[74]  Claire Petitmengin,et al.  Towards the source of thoughts The gestural and transmodal dimension of lived experience , 2007 .

[75]  Phoebe Sengers,et al.  The Three Paradigms of HCI , 2007 .

[76]  Claire Petitmengin,et al.  Describing one’s subjective experience in the second person: An interview method for the science of consciousness , 2006 .

[77]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[78]  S. Gallagher How the body shapes the mind , 2005 .

[79]  Francisco J. Varela,et al.  On Becoming Aware: A pragmatics of experiencing , 2003 .

[80]  Brad H. Story,et al.  An overview of the physiology, physics and modeling of the sound source for vowels , 2002 .

[81]  F. Varela,et al.  Radical embodiment: neural dynamics and consciousness , 2001, Trends in Cognitive Sciences.

[82]  Paul Dourish,et al.  Where the action is , 2001 .

[83]  F. Varela,et al.  The View From Within: First-Person Approaches to the Study of Consciousness , 1999 .

[84]  T. Hemsley Singing and Imagination: A Human Approach to a Great Musical Tradition , 1999 .

[85]  B. J. Fogg,et al.  Persuasive computers: perspectives and research directions , 1998, CHI.

[86]  Dag Svanæs,et al.  Kinaesthetic thinking: The tacit dimension of interaction design , 1997 .

[87]  Donald A. Schön Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions , 1987 .

[88]  J. Hines Great singers on great singing , 1982 .

[89]  James L. McClelland,et al.  Phenomenology of perception. , 1978, Science.

[90]  William J. Hardcastle,et al.  Physiology of speech production: An introduction for speech scientists , 1976 .

[91]  D. Ihde The Experience of Technology: Human-Machine Relations , 1975 .

[92]  R. Hepburn,et al.  BEING AND TIME , 2010 .