Learning source-aware representations of music in a discrete latent space

In recent years, neural network based methods have been proposed as a method that can generate representations from music, but they are not human readable and hardly analyzable or editable by a human. To address this issue, we propose a novel method to learn source-aware latent representations of music through Vector-Quantized Variational Auto-Encoder(VQ-VAE). We train our VQ-VAE to encode an input mixture into a tensor of integers in a discrete latent space, and design them to have a decomposed structure which allows humans to manipulate the latent vector in a source-aware manner. This paper also shows that we can generate bass lines by estimating latent vectors in a discrete space.