A heterogenous automatic feedback semi-supervised method for image reranking

Image reranking, which aims at enhancing the quality of keyword-based image search with the help of image features, recently has become attractive in image search community. A major challenging in this task is that image's visual features do not always well reflect image's semantic meaning. Thus, reranking methods only depending on visual features cannot guarantee to obtain good results. In addition, it is well known that the visual features of an image have strong/weak correlations with its surrounding text. Thus, it is expected that a model considering both visual features and its surrounding text can perform better than those only considering visual features. Motivated by this, in this paper, we propose the HAFSRerank--Heterogenous Automatic Feedback Semi-supervised Reranking method which makes use of both visual and textual features simultaneously during reranking. Specifically, in HAFSRerank, a multigraph is firstly constructed in which each node representing an image includes visual and textual features, and the parallel edges between them are weighted by intra-modal similarity and inter-modal similarity. A heterogenous complete graph is further derived from the multigraph. Then, an automatic feedback graph-based semi-supervised learning method is proposed to propagate the reranking scores on the complete graph, which can make use of the inter-modal similarity to update the weights of heterogenous graph automatically. Finally, the result of the semi-supervised learning is used to rerank the images. The experimental results show that HAFSRerank is superior or highly competitive to some state-of-the-art graph-based reranking methods. Moreover, the proposed reranking algorithm can be well interpreted by Bayesian theory, and does not require complex search models for special queries and any additional input from users.