Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters