Scene Retrieval Using Text-to-image GAN-based Visual Similarities and Image-to-text Model-based Textual Similarities

Scene retrieval from a video database is a fundamental study in computer vision. Traditionally, content based retrieval methods can retrieve objective scenes with high accuracy by utilizing visual features. However, users cannot utilize content based retrieval methods when they cannot prepare query contents. To solve this problem, in this paper, we propose a novel content based scene retrieval method focusing on text-to-image Generative Adversarial Network and image-to-text model. By utilizing the proposed method, we can retrieve objective scenes in visual feature space with high accuracy even though it only utilizes a sentence as an input. Experimental results show the effectiveness of the proposed method.