Tools for visual scene recognition

Scene recognition is an important step towards a full understanding of an image. This thesis presents novel ideas related to semantic-spatial content capture and local-global feature fusion techniques and applies them for scene recognition. It shows how the proper use of these approaches, without trying to recognize objects in the scene images, can lead to improvement in recognition accuracy for scene classification. First, we propose a method to build a semantic visual vocabulary. The features are extracted from the image patches and the initial vocabulary is constructed by performing k-means clustering on the extracted features and choosing the cluster centers as the visual words. The feature vectors are quantized based on the initial vocabulary to form a wordimage matrix that describes the occurrence of words in the images. The codebooks are then embedded into the concept space by latent semantic models. We demonstrate this embedding using Latent Semantic Analysis (LSA) as well as Probabilistic Latent Semantic Analysis (pLSA). In the proposed space, the distances between words represent the semantic distances, which are used to construct a discriminative and semantically meaningful vocabulary. The main contributions of the first chapter are as follows: 1. Using semantic word space to co-cluster similar words together to form a semantic visual vocabulary. This will improve the results compared to other methods that use document space directly after pLSA embedding. 2. Investigating changes in the number of latent variables. 3. Using LSA embedding when all other vision systems to date only use pLSA. This method has shown promising results on 15-Scene categories when the proposed model extracts one type of visual feature. Second, since fusing local and global features is beneficial for achieving a promising performance for scene categorization systems [7], we propose a novel Local-Global Feature Fusion (LGFF) method with the capability to fuse latent semantic patches adaptively.