Smart Musical Instruments: Vision, Design Principles, and Future Directions

Smart musical instruments (SMIs) are a family of Internet of Musical Things devices for music creation. They are characterized by sensors, actuators, embedded intelligence, and wireless connectivity to local networks and to the Internet. In this paper, we depict a vision for this recent research area, which merges the research fields on digital musical instruments and smart objects, and fosters new types of interactions between the player and the instrument, between the player and other players, and between the player and audience members. We propose a set of capabilities and technical features characterizing SMIs, as well as a design for a technological architecture supporting them. To illustrate possible applications enabled by this vision, we present a set of scenarios exploiting the intelligence embedded in SMIs. We also propose a set of guidelines that can help digital luthiers in the process of designing SMIs. Finally, we present a set of directions for future research.

[1]  Fabio Morreale,et al.  NIME Identity from the Performer’s Perspective , 2018, NIME.

[2]  lvaro Barbosa,et al.  Displaced Soundscapes: A Survey of Network Systems for Music and Sonic Art Creation , 2003, Leonardo Music Journal.

[3]  Georg Essl Automated Ad Hoc Networking for Mobile and Hybrid Music Performance , 2011, ICMC.

[4]  James A. Hendler,et al.  The Semantic Web" in Scientific American , 2001 .

[5]  Edgar Berdahl,et al.  The joystyx: a quartet of embedded acoustic instruments , 2017, NIME.

[6]  Andrew P. McPherson,et al.  The Magnetic Resonator Piano: Electronic Augmentation of an Acoustic Grand Piano , 2010 .

[7]  Baptiste Caramiaux,et al.  The Machine Learning Algorithm as Creative Musical Tool , 2016, ArXiv.

[8]  Hans Leeuw,et al.  The Electrumpet , a Hybrid Electro-Acoustic Instrument , 2009, NIME.

[9]  Cléo Palacio-Quintin,et al.  The Hyper-Flute , 2003, NIME.

[10]  Matthew Wright,et al.  Problems and prospects for intimate musical control of computers , 2001 .

[11]  Victor Lazzarini,et al.  Ecologically Grounded Creative Practices in Ubiquitous Music , 2017, Organised Sound.

[12]  Thor Magnusson,et al.  The acoustic, the digital and the body: a survey on musical instruments , 2007, NIME '07.

[13]  Ellen Campana,et al.  Amateur Musicians, Long-Term Engagement, and HCI , 2013, Music and Human-Computer Interaction.

[14]  Cumhur Erkut,et al.  Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions , 2016, Computer Music Journal.

[15]  György Fazekas,et al.  An Ontology for Audio Features , 2016, ISMIR.

[16]  Joseph A. Paradiso,et al.  Personalization, Expressivity, and Learnability of an Implicit Mapping Strategy for Physical Interfaces , 2005 .

[17]  Martin Maier,et al.  The tactile internet: vision, recent progress, and open challenges , 2016, IEEE Communications Magazine.

[18]  Georg Essl,et al.  MoveOSC - Smart Watches in Mobile Music Performance , 2014, ICMC.

[19]  M. Sile O'Modhrain,et al.  A Framework for the Evaluation of Digital Musical Instruments , 2011, Computer Music Journal.

[20]  Carman Neustaedter,et al.  Autobiographical design in HCI research: designing and learning through use-it-yourself , 2012, DIS '12.

[21]  Victor. Lazzarini,et al.  Embedded Sound Synthesis , 2015 .

[22]  Jônatas Manzolli,et al.  Dynamic Mapping Strategies Using Content-Based Classification: A Proposed Method for an Augmented Instrument , 2016, CMMR.

[23]  Daniel Overholt,et al.  The Musical Interface Technology Design Space , 2009, Organised Sound.

[24]  György Fazekas,et al.  An Overview of Semantic Web Activities in the OMRAS2 Project , 2010 .

[25]  Robert Manzke,et al.  Embedded Multichannel Linux Audiosystem for Musical Applications , 2017, Audio Mostly Conference.

[26]  Federica Cena,et al.  Principles to Design Smart Physical Objects as Adaptive Recommenders , 2017, IEEE Access.

[27]  Matthew E. P. Davies,et al.  Embedded Systems Feel the Beat in New Orleans: Highlights from the IEEE Signal Processing Cup 2017 Student Competition [SP Competitions] , 2017, IEEE Signal Process. Mag..

[28]  Nicola Orio,et al.  Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI , 2001, Computer Music Journal.

[29]  Edgar Berdahl,et al.  Advancements in Actuated Musical Instruments , 2011, Organised Sound.

[30]  Yongmeng Wu,et al.  Open Symphony: Creative Participation for Audiences of Live Music Performances , 2017, IEEE Multim..

[31]  Eleonora Borgia,et al.  The Internet of Things vision: Key features, applications and open issues , 2014, Comput. Commun..

[32]  Andrew P. McPherson,et al.  An Environment for Submillisecond-Latency Audio and Sensor Processing on BeagleBone Black , 2015 .

[33]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[34]  S. Jordà Instruments and Players: Some Thoughts on Digital Lutherie , 2004 .

[35]  Juan C. Vasquez,et al.  Idiomatic composition practices for new musical instruments: context, background and current applications , 2017, NIME.

[36]  A. Tveit,et al.  The Overtone Fiddle: an Actuated Acoustic Instrument , 2011 .

[37]  William Brent A Timbre Analysis And Classification Toolkit For Pure Data , 2010, ICMC.

[38]  Thor Magnusson,et al.  Musical Organics: A Heterarchical Approach to Digital Organology , 2017 .

[39]  Luca Turchet,et al.  Real-Time Hit Classification in a Smart Cajón , 2018, Front. ICT.

[40]  Wendy Ju,et al.  Satellite CCRMA: A Musical Interaction and Sound Synthesis Platform , 2011, NIME.

[41]  Joseph T. Chung,et al.  Hyperinstruments: Musically Intelligent and Interactive Performance and Creativity Systems , 1989, ICMC.

[42]  Perry R. Cook,et al.  Re-Designing Principles for Computer Music Controllers: a Case Study of SqueezeVox Maggie , 2009, NIME.

[43]  Luca Turchet,et al.  Demo of interactions between a performer playing a Smart Mandolin and audience members using Musical Haptic Wearables , 2018, NIME.

[44]  Douglas Schuler,et al.  Participatory Design: Principles and Practices , 1993 .

[45]  Matthew Wright,et al.  Open Sound Control: an enabling technology for musical networking , 2005, Organised Sound.

[46]  Anssi Klapuri,et al.  Automatic music transcription: challenges and future directions , 2013, Journal of Intelligent Information Systems.

[47]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[48]  Nick Bryan-Kinns,et al.  Identifying mutual engagement , 2012, Behav. Inf. Technol..

[49]  Luca Turchet,et al.  Smart Mandolin: autobiographical design, implementation, use cases, and lessons learned , 2018, Audio Mostly Conference.

[50]  Carlo Fischione,et al.  Towards the Internet of Musical Things , 2017 .

[51]  Andrew P. McPherson,et al.  Action-Sound Latency: Are Our Tools Fast Enough? , 2016, NIME.

[52]  D. Norman The Design of Everyday Things: Revised and Expanded Edition , 2013 .

[53]  Norbert Schnell,et al.  Nü Soundworks: Using spectators smartphones as a distributed network of speakers and sensors during live performances , 2017 .

[54]  Carlo Fischione,et al.  Millimeter Wave Cellular Networks: A MAC Layer Perspective , 2015, IEEE Transactions on Communications.

[55]  György Fazekas,et al.  Novel Methods in Facilitating Audience and Performer Interaction Using the Mood Conductor Framework , 2013, CMMR.

[56]  Seungmin Rho,et al.  Music Ontology for Mood and Situation Reasoning to Support Music Retrieval and Recommendation , 2009, 2009 Third International Conference on Digital Society.

[57]  Luca Turchet,et al.  Some reflections on the relation between augmented and smart musical instruments , 2018, Audio Mostly Conference.

[58]  G. Paine Towards Unified Design Guidelines for New Interfaces for Musical Expression , 2009, Organised Sound.

[59]  Mark Sandler,et al.  Enabling Interactive and Interoperable Semantic Music Applications , 2018 .

[60]  Udo Zölzer,et al.  Adaptive digital audio effects (a-DAFx): a new class of sound transformations , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[61]  Thor Magnusson,et al.  Designing Constraints: Composing and Performing with Digital Musical Systems , 2010, Computer Music Journal.

[62]  Kunal Jathal Real-Time Timbre Classification for Tabletop Hand Drumming , 2017, Computer Music Journal.

[63]  Gil Weinberg,et al.  Interconnected Musical Networks: Toward a Theoretical Framework , 2005, Computer Music Journal.

[64]  Mark B. Sandler,et al.  The Music Ontology , 2007, ISMIR.

[65]  Amar Chaudhary,et al.  Operating Systems Latency Measurement and Analysis for Sound Synthesis and Processing Applications , 1997, ICMC.

[66]  Atau Tanaka,et al.  Musical Performance Practice on Sensor-based Instruments , 2000 .

[67]  Malcolm Slaney,et al.  Semantic-audio retrieval , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[68]  Perry R. Cook,et al.  Principles for Designing Computer Music Controllers , 2001, NIME.

[69]  P. Driessen,et al.  Reconfigurable Autonomous Novel Guitar Effects ( RANGE ) , 2014 .

[70]  J. Stephen Downie,et al.  Music information retrieval , 2005, Annu. Rev. Inf. Sci. Technol..

[71]  Vesa Välimäki,et al.  Oscillator and Filter Algorithms for Virtual Analog Synthesis , 2006, Computer Music Journal.

[72]  Carlo Fischione,et al.  Smart instruments : Towards an ecosystem of interoperable devices connecting performers and audiences , 2019 .

[73]  Mathieu Barthet,et al.  Smart Musical Instruments , 2019, Foundations in Sound Design for Embedded Media.

[74]  Sherali Zeadally,et al.  Toward efficient smartification of the Internet of Things (IoT) services , 2017, Future Gener. Comput. Syst..

[75]  Carlo Fischione,et al.  Examples of use cases with Smart Instruments , 2017, Audio Mostly Conference.

[76]  Sergi Jordà,et al.  The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces , 2007, TEI.

[77]  David B. Sharp,et al.  Simulations of modal active control applied to the self-sustained oscillations of the clarinet , 2014 .

[78]  Alexandre Clément,et al.  Bridging the gap between performers and the audience using networked smartphones : the a . bel system , 2016 .

[79]  Marcelo M. Wanderley,et al.  Multiple-Model Linear Kalman Filter Framework for Unpredictable Signals , 2014, IEEE Sensors Journal.

[80]  Gerd Kortuem,et al.  Smart objects as building blocks for the Internet of things , 2010, IEEE Internet Computing.

[81]  Mahadev Satyanarayanan,et al.  The Emergence of Edge Computing , 2017, Computer.

[82]  Lauren Hayes,et al.  Towards an Aesthetics of Touch , 2017, MOCO.

[83]  Stefan Rennick Egglestone,et al.  Crafting Interactive Decoration , 2017, ACM Trans. Comput. Hum. Interact..

[84]  Vasilis Friderikos,et al.  Realizing the Tactile Internet: Haptic Communications over Next Generation 5G Cellular Networks , 2015, IEEE Wireless Communications.

[85]  Youngmoo E. Kim,et al.  Augmentation of Acoustic Drums using Electromagnetic Actuation and Wireless Control , 2018 .

[86]  Fabio Morreale,et al.  Democratising DMIs: the relationship of expertise and control intimacy , 2018, NIME.

[87]  Serge Lemouton,et al.  The augmented violin project: research, composition and performance report , 2006, NIME.

[88]  S. Finney,et al.  Auditory Feedback and Musical Keyboard Performance , 1997 .

[89]  TU MarioHermann Design Principles for Industrie 4 . 0 Scenarios , 2015 .

[90]  Marcelo M. Wanderley,et al.  The Importance of Parameter Mapping in Electronic Instrument Design , 2002, NIME.

[91]  Myles Borins,et al.  Embedded Networking and Hardware-Accelerated Graphics with Satellite CCRMA , 2013, NIME.

[92]  Luca Turchet,et al.  The Hyper-Mandolin , 2017, Audio Mostly Conference.

[93]  René Caussé,et al.  An active mute for the trombone. , 2015, The Journal of the Acoustical Society of America.

[94]  Andrew P. McPherson,et al.  Co-design of a Smart Cajón , 2018 .

[95]  L. Turchet HARD REAL-TIME ONSET DETECTION OF PERCUSSIVE SOUNDS , 2018 .

[96]  Antonella De Angeli,et al.  Musical Interface Design: An Experience-oriented Framework , 2014, NIME.

[97]  K. B. Letaief,et al.  A Survey on Mobile Edge Computing: The Communication Perspective , 2017, IEEE Communications Surveys & Tutorials.

[98]  Sang Won Lee,et al.  Mobile Devices as Musical Instruments - State of the Art and Future Prospects , 2017, CMMR.

[99]  Christopher Dobrian,et al.  The 'E' in NIME: Musical Expression with New Computer Interfaces , 2006, NIME.

[100]  Marco E. Pérez Hernández,et al.  Classifying Smart Objects using capabilities , 2014, 2014 International Conference on Smart Computing.

[101]  Marcelo M. Wanderley,et al.  New Digital Musical Instruments: Control And Interaction Beyond the Keyboard (Computer Music and Digital Audio Series) , 2006 .

[102]  G. Fitzpatrick,et al.  Design Implications for Technology-Mediated Audience Participation in Live Music , 2017 .

[103]  Eva Hornecker,et al.  Let's jam the reactable , 2013, ACM Trans. Comput. Hum. Interact..

[104]  Tommi Mikkonen,et al.  From the Internet of Things to the Internet of People , 2015, IEEE Internet Computing.

[105]  Edgar Berdahl,et al.  How to Make Embedded Acoustic Instruments , 2014, NIME.

[106]  Fabio Morreale,et al.  Design for longevity: ongoing use of instruments from nime 2010-14 , 2017, NIME.

[107]  Andrew P. McPherson,et al.  A Low-Cost Real-Time Tracking System for Violin , 2015 .

[108]  Augusto Sarti,et al.  An Overview on Networked Music Performance Technologies , 2016, IEEE Access.

[109]  Carlo Fischione,et al.  Internet of Musical Things: Vision and Challenges , 2018, IEEE Access.

[110]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[111]  Juan Manuel Cueva Lovelle,et al.  A review about Smart Objects, Sensors, and Actuators , 2017, Int. J. Interact. Multim. Artif. Intell..

[112]  Carlo Fischione,et al.  Design aspects of short-range millimeter-wave networks: A MAC layer perspective , 2015, IEEE Network.

[113]  Federica Cena,et al.  Multi-dimensional intelligence in smart physical objects , 2017, Information Systems Frontiers.

[114]  Steve Benford,et al.  Building a Maker Community Around an Open Hardware Platform , 2017, CHI.

[115]  Linda Candy,et al.  Designing and evaluating virtual musical instruments: facilitating conversational user interaction , 2008 .

[116]  Marcelo M. Wanderley,et al.  Libmapper: (a library for connecting things) , 2013, CHI Extended Abstracts.