Feature Selection and Kernel Design via Linear Programming

The definition of object (e.g., data point) similarity is critical to the performance of many machine learning algorithms, both in terms of accuracy and computational efficiency. However, it is often the case that a similarity function is unknown or chosen by hand. This paper introduces a formulation that given relative similarity comparisons among triples of points of the form object i is more like object j than object k, it constructs a kernel function that preserves the given relationships. Our approach is based on learning a kernel that is a combination of functions taken from a set of base functions (these could be kernels as well). The formulation is based on defining an optimization problem that can be solved using linear programming instead of a semidefinite program usually required for kernel learning. We show how to construct a convex problem from the given set of similarity comparisons and then arrive to a linear programming formulation by employing a subset of the positive definite matrices. We extend this formulation to consider representation/evaluation efficiency based on formulating a novel form of feature selection using kernels (that is not much more expensive to solve). Using publicly available data, we experimentally demonstrate how the formulation introduced in this paper shows excellent performance in practice by comparing it with a baseline method and a related state-of-the art approach, in addition of being much more efficient computationally.