Supporting Musical Practice Sessions Through HMD-Based Augmented Reality

Learning a musical instrument requires a lot of practice, which ideally, should be done every day. During practice sessions, students are on their own in the overwhelming majority of the time, but access to experts that support students "just-in-time" is limited. Therefore, students commonly do not receive any feedback during their practice sessions. Adequate feedback, especially for beginners, is highly important for three particular reasons: (1) preventing the acquirement of wrong motions, (2) avoiding frustration due to a steep learning curve, and (3) potential health problems that arise from straining muscles or joints harmfully. In this paper, we envision the usage of head-mounted displays as assistance modality to support musical instrument learning. We propose a modular concept for several assistance modes to help students during their practice sessions. Finally, we discuss hardware requirements and implementations to realize the proposed concepts.

[1]  Antonio Krüger,et al.  guitAR: supporting guitar learning through mobile projection , 2011, CHI Extended Abstracts.

[2]  Stephen M. Fiore,et al.  Picking Up STEAM: Educational Implications for Teaching with an Augmented Reality Guitar Learning System , 2013, HCI.

[3]  Daniel Afergan,et al.  Learn Piano with BACh: An Adaptive Learning Interface that Adjusts Task Difficulty Based on Brain State , 2016, CHI.

[4]  Shantanu Das,et al.  Music everywhere - augmented reality piano improvisation learning system , 2017, NIME.

[5]  Albrecht Schmidt,et al.  EMGuitar: Assisting Guitar Playing with Electromyography , 2018, Conference on Designing Interactive Systems.

[6]  Erwin Schoonderwaldt,et al.  MusicJacket—Combining Motion Capture and Vibrotactile Feedback to Teach Violin Bowing , 2011, IEEE Transactions on Instrumentation and Measurement.

[7]  Karola Marky,et al.  Mastering Music Instruments through Technology in Solo Learning Sessions , 2020, ArXiv.

[8]  Burkhard Wünsche,et al.  Music Education using Augmented Reality with a Head Mounted Display , 2013, AUIC.

[9]  Michael J. McGuffin,et al.  DualCAD: Integrating Augmented Reality with a Desktop GUI and Smartphone Interaction , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[10]  Serge Lemouton,et al.  The augmented violin project: research, composition and performance report , 2006, NIME.

[11]  Hideo Saito,et al.  Support system for guitar playing using augmented reality display , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[12]  Jia Zhou,et al.  Pilot Study of Piano Learning with AR Smart Glasses Considering Both Single and Paired Play , 2018, HCI.

[13]  Soonuk Seol,et al.  Real-time Recognition of Guitar Performance Using Two Sensor Groups for Interactive Lesson , 2018, Tangible and Embedded Interaction.

[14]  John A. Sloboda,et al.  The role of practice in the development of performing musicians , 1996 .

[15]  R. Krampe,et al.  Deliberate practice and elite musical performance , 1995 .

[16]  Daniel Afergan,et al.  Braahms: a novel adaptive musical interface based on users' cognitive state , 2015, NIME.

[17]  Kai Kunze,et al.  Making Regular Eyeglasses Smart , 2015, IEEE Pervasive Computing.

[18]  Robert J. Crutcher,et al.  The role of deliberate practice in the acquisition of expert performance. , 1993 .

[19]  Katja Rogers,et al.  P.I.A.N.O.: Faster Piano Learning with Interactive Projection , 2014, ITS '14.

[20]  C. Carrasco,et al.  The influence of the active range of movement of pianists' wrists on repetitive strain injury , 2003 .

[21]  Albrecht Schmidt,et al.  Chances and Challenges of Using Assistive Systems in Education , 2017, Mensch & Computer Workshopband.

[22]  Antonio Krüger,et al.  Using Mobile Projection to Support Guitar Learning , 2011, Smart Graphics.

[23]  Matthias Hoppe,et al.  Challenges and Opportunities of Mixed Reality Systems in Education , 2018, Mensch & Computer Workshopband.