A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning

Barlow Twins [Zbontar et al., 2021] is a recently proposed self-supervised learning (SSL) method that encourages similar representations between distorted variations (augmented views) of a sample, while minimizing the redundancy within the representation vector. Specifically, compared to the prior state-of-the-art SSL methods, Barlow Twins demonstrates two main properties. On one hand, its algorithm requires no explicit construction of negative sample pairs, and is not sensitive to large training batch sizes, both of which are the characteristics commonly seen in the recent non-contrastive SSL methods (e.g., BYOL [Grill et al., 2020] and SimSiam [Chen and He, 2020]). On the other hand, it avoids the reliance on symmetry-breaking network designs for distorted samples, which had been found to be crucial for these non-contrastive approaches (in order to avoid learning collapsed representations). We note that designing symmetry-breaking networks is not needed for the recent contrastive SSL methods (e.g., SimCLR [Chen et al., 2020])1. A natural question arises: what makes Barlow Twins an outlier among the existing SSL algorithms?