A unified view for unsupervised representation learning with density ratio estimation: Maximization of mutual information, nonlinear ICA and nonlinear subspace estimation

Unsupervised representation learning is one of the most important problems in machine learning. Recent promising methods are based on contrastive learning, which is often called self-supervised learning as well: Unsupervised representation learning is performed by solving a classification problem where class labels are automatically generated from unlabelled data. However, contrastive learning often relies on heuristic ideas, and therefore it is not easy to understand what contrastive learning is doing. In this paper, we emphasize that density ratio estimation is a promising goal for unsupervised representation learning, and promotes understanding to contrastive learning. Our primal contribution is to theoretically show that density ratio estimation unifies three frameworks for unsupervised representation learning: Maximization of mutual information (MI), nonlinear independent component analysis (ICA) and a novel framework for estimation of a lower-dimensional nonlinear subspace proposed in this paper. This unified view clarifies under what conditions contrastive learning can be regarded as maximizing MI, performing nonlinear ICA or estimating the lowerdimensional nonlinear subspace in the proposed framework. Furthermore, we also make theoretical contributions in each of the three frameworks: We show that MI for data representation can be maximized through density ratio estimation under certain conditions, while our analysis for nonlinear ICA reveals a novel insight for recovery of the latent source components, which is clearly supported by numerical experiments. In addition, the proposed new framework for nonlinear subspace estimation can be seen as a generalization of nonlinear ICA, and some theoretical conditions are established to estimate the nonlinear subspace. The unified view through density ratio estimation is useful from the practical side as well because applying a number of methods for density ratio estimation may automatically lead to practical methods for unsupervised representation learning. Following this idea, we propose two practical methods for unsupervised representation learning through density ratio estimation: The first method is an outlier-robust method for representation learning, while the second one is a sample-efficient nonlinear ICA method. Then, we theoretically investigate outlier-robustness of the proposed methods. Finally, we numerically demonstrate usefulness of the proposed methods in nonlinear ICA and through application to a downstream task for linear classification.