A General Cost Model for Dimensionality Reduction in High Dimensional Spaces

Similarity search usually encounters a serious problem in the high dimensional space, known as the "curse of dimensionality". In order to speed up the retrieval efficiency, previous approaches usually reduce the dimensionality of the entire data set to a fixed lower value before building indexes (referred to as global dimensionality reduction (GDR)). More recent works focus on locally reducing the dimensionality of data to different values (called the local dimensionality reduction (LDR)). However, so far little work has formally evaluated the effectiveness and efficiency of both GDR and LDR for range queries. Motivated by this, in this paper, we propose a general cost model for both GDR and LDR, in light of which we introduce a novel LDR method, PRANS. It can achieve high retrieval efficiency with the guarantee of optimality given by the formal model. Finally, a B+-tree index is constructed over the reduced partitions for fast similarity search. Extensive experiments validate the correctness of our cost model on both real and synthetic data sets, and demonstrate the efficiency and effectiveness of the proposed PRANS method.