MIRROR DESCENT FOR METRIC LEARNING

Introduction. The concepts of similarity, distance or metric are central to a many well-known and popular algorithms such as k-means clustering [13], the nearest neighbor algorithm [6], locallylinear embedding [14], multi-dimensional scaling [7] and semi-supervised clustering [18]. While there are many approaches to metric learning, a large body of work is focussed on learning the Mahalanobis distance, which amounts to learning a transformation and computing the distance in the transformed space. Among these approaches are the work of Xing et al., [20], relevant components analysis [17], the large-margin nearest neighbor (LMNN) algorithm [19], Globerson and Roweis’ method of collapsing classes [10], information-theoretic metric learning (ITML) [8] and BoostMetric [16]. Aside from the batch approaches above, online algorithms such as the online ITML algorithm [8] and the pseudo-metric online learning algorithm (POLA) [15] have proven successful. All these approaches are characterized by diverse loss functions and projection methods, which naturally begs the question: is there a wider framework that can generalize many of these existing methods? In addition, ever persistent issues are t hose of scalability to large data sets and the question of kernelizability. Thus, we propose a unifi ed approach to Mahalanobis metric learning: an online regularized metric learningalgorithm based on the ideas of composite objective mirror descent(COMID) [9]. We propose to formulate the metric learning problem as a regularized positive semi-definite matrix learningproblem, whose update rules can be derived using the COMID framework. This approach aims to be scalable, kernelizable, and admissible to many different types of Bregman and loss functions which allows for the tailoring of several different classes of algorithms. The most novel contribution is the use of the trace norm, which yields a sparse metric in its eigenspectrum, thus simultaneously performing feature selection along with me tric learning. Unifying Framework. The goal is to incrementally learn a squared Mahalanobis metric d(x, z) 2 = (xt −zt) ′ M(xt −zt), given training data of the form (xt, zt, yt) T=1 , where labels yt = ±1 indicate