Constraint selection in metric learning

Abstract A number of machine learning and knowledge-based algorithms are using a metric, or a distance, in order to compare individuals. The Euclidean distance is usually employed, but it may be more efficient to learn a parametric distance such as Mahalanobis metric. Learning such a metric is a hot topic since more than ten years now, and a number of methods have been proposed to efficiently learn it. However, the nature of the problem makes it quite difficult for large scale data, as well as data for which classes overlap. This paper presents a simple way of improving accuracy and scalability of any iterative metric learning algorithm, where constraints are obtained prior to the algorithm. The proposed approach relies on a loss-dependent weighted selection of constraints that are used for learning the metric. Using the corresponding dedicated loss function, the method clearly allows to obtain better results than state-of-the-art methods, both in terms of accuracy and time complexity. Some experimental results on real world, and potentially large, datasets are demonstrating the effectiveness of our proposition.

[1]  Andrew McCallum,et al.  Toward Optimal Active Learning through Monte Carlo Estimation of Error Reduction , 2001, ICML 2001.

[2]  Inderjit S. Dhillon,et al.  Metric and Kernel Learning Using a Linear Transformation , 2009, J. Mach. Learn. Res..

[3]  Haiyan Chen,et al.  Bagging-like metric learning for support vector regression , 2014, Knowl. Based Syst..

[4]  Bin Li,et al.  Sparse Online Relative Similarity Learning , 2015, 2015 IEEE International Conference on Data Mining.

[5]  A. Tversky.  Features of Similarity , 1977 .

[6]  Liwei Wang,et al.  On learning with dissimilarity functions , 2007, ICML '07.

[7]  Kilian Q. Weinberger,et al.  Large Margin Multi-Task Metric Learning , 2010, NIPS.

[8]  Silvio Savarese,et al.  Deep Metric Learning via Lifted Structured Feature Embedding , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Bernt Schiele,et al.  Active Metric Learning for Object Recognition , 2012, DAGM/OAGM Symposium.

[10]  Tat-Seng Chua,et al.  An efficient sparse metric learning in high-dimensional space via l1-penalized log-determinant regularization , 2009, ICML '09.

[11]  Francisco Herrera,et al.  Integrating Instance Selection, Instance Weighting, and Feature Weighting for Nearest Neighbor Classifiers by Coevolutionary Algorithms , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[12]  Koby Crammer,et al.  Online Passive-Aggressive Algorithms , 2003, J. Mach. Learn. Res..

[13]  Marc Sebban,et al.  Metric Learning , 2015, Metric Learning.

[14]  Inderjit S. Dhillon,et al.  Information-theoretic metric learning , 2006, ICML '07.

[15]  Gabriela Csurka,et al.  Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost , 2012, ECCV.

[16]  Yuan Shi,et al.  Sparse Compositional Metric Learning , 2014, AAAI.

[17]  Maria-Florina Balcan,et al.  On a theory of learning with similarity functions , 2006, ICML.

[18]  Alexandros Kalousis,et al.  Parametric Local Metric Learning for Nearest Neighbor Classification , 2012, NIPS.

[19]  Hoel Le Capitaine,et al.  A Relevance-Based Learning Model of Fuzzy Similarity Measures , 2012, IEEE Transactions on Fuzzy Systems.

[20]  Rong Jin,et al.  Regularized Distance Metric Learning: Theory and Algorithm , 2009, NIPS.

[21]  Gal Chechik,et al.  Learning Sparse Metrics, One Feature at a Time , 2015, FE@NIPS.

[22]  Lei Wang,et al.  Positive Semidefinite Metric Learning Using Boosting-like Algorithms , 2011, J. Mach. Learn. Res..

[23]  Prateek Jain,et al.  Similarity-based Learning via Data Driven Embeddings , 2011, NIPS.

[24]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[25]  Peng Li,et al.  Distance Metric Learning with Eigenvalue Optimization , 2012, J. Mach. Learn. Res..

[26]  Brian Kulis,et al.  Metric Learning: A Survey , 2013, Found. Trends Mach. Learn..

[27]  Byoung-Tak Zhang,et al.  Generative Local Metric Learning for Nearest Neighbor Classification , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  H. Karimi,et al.  LogDet Divergence based Metric Learning using Triplet Labels , 2013 .

[29]  Baba C. Vemuri,et al.  A Robust and Efficient Doubly Regularized Metric Learning Approach , 2012, ECCV.

[30]  Xiang Zhang,et al.  Metric Learning from Relative Comparisons by Minimizing Squared Residual , 2012, 2012 IEEE 12th International Conference on Data Mining.

[31]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[32]  Yuan-Fang Wang,et al.  Learning a Mahalanobis Distance-Based Dynamic Time Warping Measure for Multivariate Time Series Classification , 2016, IEEE Transactions on Cybernetics.

[33]  Samy Bengio,et al.  Large Scale Online Learning of Image Similarity Through Ranking , 2009, J. Mach. Learn. Res..

[34]  Rong Jin,et al.  Distance Metric Learning: A Comprehensive Survey , 2006 .

[35]  Bin Yu,et al.  Boosting with early stopping: Convergence and consistency , 2005, math/0508276.

[36]  Amir Globerson,et al.  Metric Learning by Collapsing Classes , 2005, NIPS.

[37]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[38]  Eric O. Postma,et al.  Dimensionality Reduction: A Comparative Review , 2008 .

[39]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[40]  Deepa Paranjpe,et al.  Semi-supervised clustering with metric learning using relative comparisons , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[41]  Masashi Sugiyama,et al.  Local Fisher discriminant analysis for supervised dimensionality reduction , 2006, ICML.

[42]  Hoel Le Capitaine,et al.  Block similarity in fuzzy tuples , 2013, Fuzzy Sets Syst..

[43]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.