On the consistency and features of image similarity

Image indexing and retrieval systems mostly rely on the computation of similarity measures between images. This notion is ill-defined, generally based on simplistic assumptions that do not fit the actual context of use of image retrieval systems. This paper addresses two fundamental issues related to image similarity: checking whether the degree of similarity between two images is perceived consistently by different users and establishing the elements of the images on which users base their similarity judgment. A study is set up, in which human subjects have been asked to assess the degree of the pairwise similarity of images and describe the features on which they base their judgments. The quantitative analysis of the similarity scores reported by the subjects shows that users reach a certain consensus on similarity assessment. From the qualitative analysis of the transcripts of the records of the experiments, a list of the features used by the subjects to assess image similarity is built. From this, a new model of image description emerges. As compared to existing models, it is more realistic, free of preconceptions and more suited to the task of similarity computation. These results are discussed from the perspectives of psychology and computer science.