Gram matrix based representation for image retrieval

In the field of image retrieval, most of image representations based on convolutional neural network (CNN) are first-order forms, i.e., the pooling or encoding methods are adopted on feature maps directly to produce compact image representations, while the high-order representations, such as the dependencies between different channels in the same layer are often neglected. In this paper, a novel image representation and retrieval algorithm based on Gram matrix is proposed. Specifically, based on Gram matrix of convolutional layers, second-order features are firstly constructed by considering the relationships between different channels of feature maps. Afterwards, two weighted schemes, that is, equal channel weighting and sparsity-sensitive channel weighting are presented respectively to aggregate them into the final representation. The extensive experiments on four public image datasets are conducted, and the promising results demonstrate the effectiveness of the proposed algorithm.