Face Clustering: Representation and Pairwise Constraints

Clustering face images according to their latent identity has two important applications: 1) grouping a collection of face images when no external labels are associated with images, and 2) indexing for efficient large scale face retrieval. The clustering problem is composed of two key parts: representation and similarity metric for face images, and choice of the partition algorithm. We first propose a representation based on ResNet, which has been shown to perform very well in image classification problems. Given this representation, we design a clustering algorithm, Conditional Pairwise Clustering (ConPaC), which directly estimates the adjacency matrix only based on the similarities between face images. This allows a dynamic selection of number of clusters and retains pairwise similarities between faces. ConPaC formulates the clustering problem as a Conditional Random Field model and uses Loopy Belief Propagation to find an approximate solution for maximizing the posterior probability of the adjacency matrix. Experimental results on two benchmark face datasets (LFW and IJB-B) show that ConPaC outperforms well known clustering algorithms such as <inline-formula> <tex-math notation="LaTeX">${k}$ </tex-math></inline-formula>-means, spectral clustering, and approximate Rank-order. Additionally, our algorithm can naturally incorporate pairwise constraints to work in a semi-supervised way that leads to improved clustering performance. We also propose a <inline-formula> <tex-math notation="LaTeX">${k}$ </tex-math></inline-formula>-NN variant of ConPaC, which has a linear time complexity given a <inline-formula> <tex-math notation="LaTeX">${k}$ </tex-math></inline-formula>-NN graph, suitable for large datasets.