Improving Self-Supervised Speech Representations by Disentangling Speakers