Large-scale interactive exploratory visual search

Large scale visual search has been one of the challenging issues in the era of big data. It demands techniques that are not only highly effective and efficient but also allow users conveniently express their information needs and refine their intents. In this thesis, we focus on developing an exploratory framework for large scale visual search, since users often have difficulty in clearly formulating the intent-specific queries for visual content, unlike keyword based textual information retrieval. Towards this goal, we also develop a number of enabling techniques in this thesis, including compact visual content representation for scalable search, near duplicate video shot detection, and action based event detection at a fine level. Although the bag-of-visual-words model has been successfully utilized for scalable visual information retrieval, the large number of visual words obtained from a query image results in a large amount of data to be uploaded to search engines, which significant affects the response time of a query. This problem becomes more serious for mobile users due to limited bandwidth while the mobile and cloud architecture has become more and more popular. Therefore, we propose a novel scheme for extremely low bit rate visual search, which sends compressed visual words consisting of vocabulary tree histogram and descriptor orientations rather than descriptors. This scheme can reduce the bit rate of a visual query with few extra computational costs on a mobile client. Specifically, we store a vocabulary tree and extract visual descriptors on the mobile client. A light-weight pre-retrieval is performed to obtain the visited leaf nodes in the vocabulary tree. Compact representation of video data is achieved through identifying keyframes of a video which can also help users comprehend visual content efficiently. We propose a novel Bag-of-Importance (BoI) model for static video summarization by identifying the frames with important local features as keyframes. In addition, we propose to learn a transformation from a raw local feature to a more powerful sparse non-linear representation for deriving the importance score of each local feature, rather than directly utilize the hand-crafted visual features like most of the existing approaches.