Learning from search engine and human supervision for web image search

Visual reranking aims at improving the precision of text-based Web image search. In this paper we propose to combine two learning strategies for deriving the reranking model: learning from search engine and learning from human supervision. The first strategy learns the reranking model in a pseudo-supervised fashion by interpreting parts of the initial text-based search result as pseudo-relevant. The second strategy involves manual relevance labeling of the text-based search results obtained for a limited number of representative queries. While learning from search engine is query dependent and can therefore adapt better to individual queries, it is essentially unsupervised and noisy. While human supervision can better relate the search results to true relevance criteria, it needs to be deployed in a way to keep the reranking scalable. A combination of the two is expected to benefit from their respective advantages and reduce the impact of their individual deficiencies. We propose a two-stage learning approach to visual reranking, where in the online stage multiple query-relative meta rerankers are learned in a pseudo-supervised fashion from the search results and in the offline stage human supervision is used to derive the final reranking function based on these meta rerankers. The experimental results demonstrate that the proposed method significantly outperforms the existing reranking approaches.