LeMo: Supporting Collaborative Music Making in Virtual Reality

When engaged in music composition and improvisation, composers who are working together may want to communicate in modalities other than sound since that is the primary medium of the creative activity. This raises questions about the Sonic Interaction Design of Virtual Environments, specifically in terms of how to design user experiences which support collaboration without being detrimental to the product actually being created. In this paper we explore a multimodal approach to supporting creativity in collaborative music making in Virtual Reality. We outline the design and study of our two-person Virtual Reality step sequencer named LeMo, in which people are enabled to communicate via visual representations including free-form 3D annotations instead of spoken communication, leaving their full auditory sense to experiencing the joint creation of music. We studied how people used the annotations in LeMo to support their composition process and human-human interaction and identified five classes of use of annotation, three of which are particularly relevant to the future design of Sonic Interactions in Virtual Environments.

[1]  Ge Wang,et al.  Designing Smule's Ocarina: The iPhone's Magic Flute , 2009, NIME.

[2]  Marc Leman,et al.  A multimodal interactive installation for collaborative music making : from preservation to enhanced user design , 2017 .

[3]  Sarel Lavy,et al.  A Multiuser Shared Virtual Environment for Facility Management , 2016 .

[4]  Nick Bryan-Kinns,et al.  The impact of transitions on user experience in virtual reality , 2017, 2017 IEEE Virtual Reality (VR).

[5]  Kouichi Matsuda,et al.  Virtual Society: Collaboration in 3D Spaces on the Internet , 1997, Computer Supported Cooperative Work (CSCW).

[6]  Nick Bryan-Kinns,et al.  Analysing Group Creativity: A Distributed Cognitive Study of Joint Music Composition , 2006 .

[7]  Steve Benford,et al.  User embodiment in collaborative virtual environments , 1995, CHI '95.

[8]  Mel Slater,et al.  Depth of Presence in Virtual Environments , 1994, Presence: Teleoperators & Virtual Environments.

[9]  lvaro Barbosa,et al.  Displaced Soundscapes: A Survey of Network Systems for Music and Sonic Art Creation , 2003, Leonardo Music Journal.

[10]  Carrie Heeter,et al.  Being There: The Subjective Experience of Presence , 1992, Presence: Teleoperators & Virtual Environments.

[11]  Robert K. Yin,et al.  Case Study Research and Applications: Design and Methods , 2017 .

[12]  Nick Bryan-Kinns,et al.  Annotating Distributed Scores for Mutual Engagement in Daisyphone and Beyond , 2011, Leonardo Music Journal.

[13]  Nick Bryan-Kinns,et al.  Hands where we can see them! Investigating the impact of gesture size on audience perception , 2017, ICMC.

[14]  Anderson Maciel,et al.  Using Immersive Virtual Reality to Reduce Work Accidents in Developing Countries , 2016, IEEE Computer Graphics and Applications.

[15]  Mel Slater,et al.  Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[16]  Michael J. Singer,et al.  Measuring Presence in Virtual Environments: A Presence Questionnaire , 1998, Presence.

[17]  Tina Blaine,et al.  Contexts of Collaborative Musical Experiences , 2003, NIME.

[18]  Eva Hornecker,et al.  Let's jam the reactable , 2013, ACM Trans. Comput. Hum. Interact..

[19]  Chris Greenhalgh,et al.  The DiveBone—an application-level network architecture for Internet-based CVEs , 1999, VRST '99.

[20]  P. Wallace,et al.  The Impact of Avatar Self-Representation on Collaboration in Virtual Worlds , 2009 .

[21]  Ivan Poupyrev,et al.  Mixing realities in Shared Space: an augmented reality interface for collaborative computing , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[22]  Mel Slater,et al.  A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments , 1997, Presence: Teleoperators & Virtual Environments.

[23]  Nick Bryan-Kinns,et al.  Inter-Play : Understanding Group Music Improvisation as a Form of Everyday Interaction , 2005 .

[24]  George W. Furnas,et al.  The effectiveness of multiscale collaboration in virtual environments , 2003, CHI Extended Abstracts.

[25]  D. Romano,et al.  Collaborative Decision-Making and Presence in Shared Dynamic Virtual Environments , 1998 .

[26]  Steve Benford,et al.  Collaborative virtual environments , 2001, CACM.

[27]  Andrew E. Johnson,et al.  NICE: combining constructionism, narrative and collaboration in a virtual learning environment , 1997, COMG.

[28]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[29]  Nick Bryan-Kinns.  Daisyphone: the design and impact of a novel environment for remote group music improvisation , 2004, DIS '04.

[30]  Nasser Nassiri,et al.  Human interactions and personal space in collaborative virtual environments , 2010, Virtual Reality.

[31]  Tom Rodden,et al.  A Survey of CSCW Systems , 1991, Interact. Comput..

[32]  Nick Bryan-Kinns,et al.  Identifying mutual engagement , 2012, Behav. Inf. Technol..

[33]  Nick Bryan-Kinns,et al.  Drawing Electroacoustic Music , 2008, ICMC.