Learning query and image similarities with listwise supervision

One of the fundamental problems in image search is to learn the ranking functions, i.e., similarity between the textual query and visual image. A number of research paradigms, ranging from feature-based vector model to image ranker learning, have been applied to measure query-image similarity. However, most of the existing similarity learning methods either depend on surrounding texts for ranking images or learn image rankers to satisfy pairwise or tripletwise supervision. In this paper, we propose to leverage listwise supervision into a principled click-through-based query-image similarity learning framework. In particular, the algorithm utilizes click counts for each image in response to a query to get a ranking list. The ranking information is represented by a set of rank triplets that can be used to assess the quality of the ranking list. The image ranking problem is then solved efficiently by learning two linear projections for query and image space respectively, through maximizing the ranking quality over all the training data. When the two linear projections are learnt, query-image similarity can be directly computed by dot product on this projected subspace. On a large-scale click-through-based image dataset with 11.7 million queries and one million images, our learnt model via listwise supervision is shown to be powerful for keyword-based image search with superior performance over several state-of-the-art methods.