A Unified Framework for Contrastive Learning from a Perspective of Affinity Matrix