Optimally Extracting Discriminative Disjunctive Features for Dimensionality Reduction

Dimension Reduction is one popular approach to tackle large and redundant feature spaces as seen in most practical problems, either by selecting a subset of features or by projecting the features onto a smaller space. Most of these approaches suffer from the drawback that the dimensionality reduction objective and the objective for classifier training are decoupled. Recently, there have been some efforts to address the two tasks in a combined manner by attempting to solve an upper-bound to a single objective function. But the main drawback of these methods is that they are all parametric, in the sense that the number of reduced dimensions needs to be provided as an input to the system. Here we propose an integrated non-parametric learning approach to supervised dimension reduction by exploring a search space of all possible disjunctions of features and discovering a sparse subset of (interpretable) disjunctions that minimise a regularised loss function. Here, in order to discover good disjunctive features, we employ algorithms from hierarchical kernel learning to simultaneously achieve efficient feature selection and optimal classifier training in a maximum margin framework and demonstrate the effectiveness of our approach on benchmark datasets.