Modeling conventional and hyperspectral image-sets for classification

Traditional image classification algorithms were designed to classify single test images. However, in many practical applications, multiple images of the query object are available. Image-set modeling aims to efficiently represent and classify a collection of images that belong to the same class. Classification based on image-sets is more attractive because the appearance variations, due to changes in pose, illumination and scale of an object or even a scene can be captured in multiple images of the set. Image set modeling aims at explicitly modeling these variations to achieve better classification accuracy. Classification based on image-sets must essentially address two core challenges; how to effectively model the intra-class appearance variations using a robust representation and how to define a distance measure that exploits the inter-class variations within the set representation. This thesis proposes efficient and accurate representations to model conventional and hyperspectral image-sets. Several contributions are made with emphasis on designing efficient algorithms that learn the complex nonlinear image-set structures without making prior assumptions about the underlying images or their distributions. In the conventional category, this thesis deals with grayscale, colour (RGB) and near infrared images. Hyperspectral images are basically images acquired at a large number of narrow bands of the visible spectrum and beyond. In the hyperspectral category, images comprising 33 to 65 bands in the wavelength range of 400-1000nm are used. Hyperspectral images are modeled as image-sets for the first time in this research. Two methods are proposed for representing hyperspectral image-sets. The first one fuses the spatiospectral mean with the covariance to represent a hyperspectral image cube as a compact feature vector. Fusion is performed by sliding a cubelet over the hyperspectral image cube and integrating the first and second order statistics of the local neighbourhood. This approach minimizes the effects of inter band misalignments that are unavoidable due to the sequential capture of hyperspectral bands. The second method jointly models the rich spatiospectral information of the hyperspectral image-set and is based on the three dimensional Discrete Cosine Transform (3D-DCT). It represents a hyperspectral image cube with a few low frequency 3D-DCT coefficients. Classification is performed using Partial Least Squares regression in both cases. Both representations are evaluated on the task of hyperspectral face recognition on three standard datasets. One of these datasets was acquired as a part of this thesis. Comparisons with grayscale and RGB image based face recognition algorithms show that hyperspectral images provide improved accuracy. This thesis also performs a detailed study on whether the spectral reflectance of the face alone can be used as a reliable biometric. Conventional images contain more variations due to changes in pose and appearance of the objects in a given class. Three methods are presented in this thesis for conventional image-set modeling. The first method models image sets with the mean and Cholesky decomposed global covariance descriptor. By projecting the mean shifted covariance features to a low dimensional subspace, a compact and discriminative representation is obtained. The second approach performs image-set modeling without making prior assumptions about the image-set data. Image-sets are modeled with Deep Extreme Learning Machines (DELM) which are neural networks with extremely fast training time and generalize well even when trained on a limited number of samples. These models automatically learn the nonlinear structure of the manifold on which the image-set data lie. The proposed DELM based image-set modeling can scale to extremely large image-set databases. The third approach obtains a single robust image-set representation by a discriminative and sparse combination of multiple image-set representations. For this purpose, a sparse kernel learning (SKL) algorithm is proposed and formulated as a sparse linear discriminant analysis based objective function to learn the most discriminative linear combination of the image-set kernels. Such an approach is robust to the weaknesses of individual image-set representations. SKL is generic and can be applied to a combination of existing image-set kernels and new ones proposed in the future. The final contribution of this thesis is a blind domain adaptation algorithm. In practical applications, the test data often have different distribution from the training data leading to suboptimal visual classification performance. Domain adaptation addresses this problem by designing classifiers that are robust to mismatched distributions. The proposed domain adaptation algorithm does not require target domain samples for training. During training a global nonlinear Extreme Learning Machine (ELM) model is learned from the source domain data in an unsupervised fashion. During testing, the target domain features are augmented with the reconstructed features from the global ELM model. The resulting enriched features are then classified using the class specific ELM models based on minimum reconstruction error. Extensive experiments are performed on a range of benchmark databases for imageset based face recognition, periocular recognition, object categorization and hand gesture recognition. The proposed algorithms are thoroughly evaluated on the benchmark datasets and compared with a large number of existing state-of-the-art algorithms. The proposed algorithms demonstrate improved classification accuracy over the existing algorithms. Most importantly, the proposed algorithms are significantly more efficient in terms of execution time and memory requirement.