Witnessing the build-up of the colour-density relation