An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation

Music relies heavily on self-reference to build structure and meaning. We explore the Transformer architecture [27] as a generative model for music, as self-attention has shown compelling results on tasks that require long-term structure such as Wikipedia summary generation [18]. However, timing information is critical for polyphonic music, and Transformer does not explicitly model absolute or relative timing in its structure. To address this challenge, Shaw et al. [22] introduced relative position representations to self-attention to improve machine translation. However, the formulation was not scalable to longer sequences. We propose an improved formulation which reduces the memory requirements of the relative position computation from O(ld) to O(ld), where l is the length of sequences and d is the hidden size, making it possible to train much longer sequences and achieve faster convergence. In experiments on symbolic music we find that relative selfattention substantially improves sample quality for unconditioned generation and is able to generate sequences of lengths longer than those from the training set. When primed with an initial sequence, the model generates continuations that develop the prime coherently and exhibit long-term structure. Relative self-attention can be instrumental in capturing richer relationships within a musical piece 2 3.