Beyond Sum and Weighted Aggregation: An Efficient Mixed Aggregation Method with Multiple Weights for Image Search

Image search with local descriptors represents an image usually by embedding and aggregating a set of patch descriptors into a single vector. Standard aggregation operations include sum and weighted aggregations. While showing high efficiency, sum aggregation lacks discriminative power. In contrast, weighted aggregation shows promising retrieval performance but suffers extremely high time cost. In this paper, we present a general mixed aggregation method that unifies sum and weighted aggregation methods. Owing to its general formulation, our method is able to balance the trade-off between quality and speed. Furthermore, we propose to compute multiple weighting coefficients rather than one for each to be aggregated vector by partitioning it into several components. Experimental results demonstrate that, while showing over ten times speedup over baselines, the image search frameworks with our mixed aggregation method achieve the state-of-the-art performance. Inspired by our aggregation method, we also present a new embedding strategy. Different from the existing embedding methods that individually map each descriptor into a single embedded vector, our embedding method maps a group of local descriptors into a single vector, which significantly benefits the aggregation step in terms of speed. As demonstrated by the experiments, the retrieval frameworks with our embedding method are more than fifty times faster than baselines, while maintaining competitive retrieval performance.