Image retrieval using image context vectors: first results

The key question for any image retrieval approach is how to represent the images. We are exploring a new image context vector representation that avoids the need for full image understanding. This representation: (1) is invariant with respect to translation and scaling of (whole) images, (2) is robust with respect to translation, scaling, small rotations, and partial occlusions of objects within images, (3) avoids explicit segmentation into objects, and (4) allows computation of image-query similarity using only about 300 multiplications and additions. A context vector is a high (approximately 300) dimensional vector that can represent images, subimages, or image queries. Image context vectors are an extension of previous work in document retrieval where context vectors were used to represent documents, terms, and queries. The image is first represented as a collection of pairs of features. Each feature pair is then transformed into a 300-dimensional context vector that encodes the feature pair and its orientation. All the vectors for pairs are added together to form the context vector for the entire image. Retrieval order is determined by taking dot products of image context vectors with a query context vector, a fast operation. Results from a first prototype look promising. 119