VirtuosoNet: A Hierarchical RNN-based System for Modeling Expressive Piano Performance

In this paper, we present our application of deep neural network to modeling piano performance, which imitates the expressive control of tempo, dynamics, articulations and pedaling from pianists. Our model consists of recurrent neural networks with hierarchical attention and conditional variational autoencoder. The model takes a sequence of note-level score features extracted from MusicXML as input and predicts piano performance features of the corresponding notes. To render musical expressions consistently over long-term sections, we first predict tempo and dynamics in measure-level and, based on the result, refine them in note-level. The evaluation through listening test shows that our model achieves a more human-like expressiveness compared to previous models. We also share the dataset we used for the experiment.

[1]  Carlos Eduardo Cancino-Chacón,et al.  The Basis Mixer : A Computational Romantic Pianist , 2016 .

[2]  Eita Nakamura,et al.  Performance Error Detection and Post-Processing for Fast and Accurate Symbolic Music Alignment , 2017, ISMIR.

[3]  Diyi Yang,et al.  Hierarchical Attention Networks for Document Classification , 2016, NAACL.

[4]  J. Sundberg,et al.  Overview of the KTH rule system for musical performance. , 2006 .

[5]  Sergio Canazza,et al.  Algorithms can Mimic Human Piano Performance: The Deep Blues of Music , 2017 .

[6]  Juhan Nam,et al.  Music Encoding Conference 2019 Score and Performance Features Score and Performance Features for Rendering Expressive Music Performances , 2019 .

[7]  Colin Raffel,et al.  A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music , 2018, ICML.

[8]  Gerhard Widmer,et al.  An evaluation of linear and non-linear models of expressive dynamics in classical piano and symphonic music , 2017, Machine Learning.

[9]  Satoru Fukayama,et al.  Statistical Approach to Automatic Expressive Rendition of Polyphonic Piano Music , 2013, Guide to Computing for Expressive Music Performance.

[10]  Gerhard Widmer,et al.  Computational Models of Expressive Music Performance: A Comprehensive and Critical Review , 2018, Front. Digit. Humanit..

[11]  Douglas Eck,et al.  Music Transformer , 2018, 1809.04281.

[12]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[13]  Samy Bengio,et al.  Generating Sentences from a Continuous Space , 2015, CoNLL.

[14]  Douglas Eck,et al.  Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset , 2018, ICLR.

[15]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[16]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[17]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[18]  Douglas Eck,et al.  This time with feeling: learning expressive musical performance , 2018, Neural Computing and Applications.

[19]  Sergio Giraldo,et al.  A machine learning approach to ornamentation modeling and synthesis in jazz guitar , 2016, Machine Learning and Music Generation.

[20]  Gerhard Widmer,et al.  YQX Plays Chopin , 2009, AI Mag..

[21]  Geraint A. Wiggins,et al.  Linking melodic expectation to expressive performance timing and perceived musical tension. , 2016, Journal of experimental psychology. Human perception and performance.

[22]  Douglas Eck,et al.  Learning to Groove with Inverse Sequence Transformations , 2019, ICML.

[23]  Stanislas Lauly.  Modélisation de l'interprétation des pianistes & applications d'auto-encodeurs sur des modèles temporels , 2010 .

[24]  Gerhard Widmer,et al.  The Magaloff Project: An Interim Report , 2010 .

[25]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[26]  Juhan Nam,et al.  Graph Neural Network for Music Score Data and Modeling Expressive Piano Performance , 2019, ICML.

[27]  Carl Henrik Ek,et al.  Neural Translation of Musical Style , 2017, ArXiv.

[28]  Sergio Canazza,et al.  CaRo 2.0: An Interactive System for Expressive Music Rendering , 2015, Adv. Hum. Comput. Interact..

[29]  Yoshua Bengio,et al.  Hierarchical Multiscale Recurrent Neural Networks , 2016, ICLR.