Improving image similarity estimation via global distance distribution information

Abstract Estimating the similarity between two images or image patches is at the heart of many computer vision problems including content-based image retrieval, image registration, and scene recognition. However, commonly used distance-based similarity estimation is not always reliable due to the limitations in both image understanding techniques and distance metrics. In this paper, we present a scheme to improve the similarity estimation under image search scenario. To this end, we explore the discriminative capability underlying global distance distribution obtained by querying an auxiliary image dataset in an unsupervised manner. According to the results of motivational experiments, we discover that global distance distributions have the desired capability in distinguishing inter-class images which can be applied to enhance the original distance metric. Following this finding, we propose a novel approach to incorporate the global distance distribution into the original distance metric to improve the reliability of the similarity estimation. One key novelty of this approach is to model the global distance distribution as Rayleigh distribution and then represent the difference between two distributions by the relative entropy. In this way, the difference between two global distance distributions can be calculated in an extremely efficient way. We also demonstrate that Rayleigh distribution leads to consistent performance compared to the real distribution. Extensive experiments on three public datasets with various image representations and distance metrics show that the enhanced similarity estimation remarkably outperforms the original one. Furthermore, the proposed approach shows the desired scalability for handling large-scale image search scenarios.