Internet-Based Image Retrieval Using End-to-End Trained Deep Distributions

Internet image search engines have long been considered as a promising tool for handling open-vocabulary textual user queries to unannotated image datasets. However, systems that use this tool have to deal with multi-modal and noisy image sets returned by search engines, especially for polysemous queries. Generally, for many queries, only a small part of the returned sets can be relevant to the user intent. In this work, we suggest an approach that explicitly accounts for the complex and noisy structure of the image sets returned by Internet image search engines. Similarly to a considerable number of previous image retrieval works, we train a deep convolutional network that maps images to high-dimensional descriptors. To model image sets obtained from the Internet, our approach then fits a simple probabilistic model that accounts for multi-modality and noise (e.g. a Gaussian mixture model) to the deep descriptors of the images in this set. Finally, the resulting distribution model can be used to search in the unannotated image dataset by evaluating likelihoods of individual images. As our main contribution, we develop an end-to-end training procedure that tunes the parameters of a deep network using an annotated training set, while accounting for the distribution fitting and the subsequent matching. In the experiments, we show that such an end-to-end approach boosts the accuracy of the Internet-based image retrieval for hold-out concepts, as compared to retrieval systems that fit similar distribution models to pre-trained features and to simpler end-to-end trained baselines.