Metric Learning-Guided Least Squares Classifier Learning

For a multicategory classification problem, discriminative least squares regression (DLSR) explicitly introduces an <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula>-dragging technique to enlarge the margin between the categories, yielding superior classification performance from a margin perspective. In this brief, we reconsider this classification problem from a metric learning perspective and propose a framework of metric learning-guided least squares classifier (MLG-LSC) learning. The core idea is to learn a unified metric matrix for the error of LSR, such that such a metric matrix can yield small distances for the same category, while large ones for the different categories. As opposed to the <inline-formula> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula>-dragging in DLSR, we call this the error-dragging (e-dragging). Different from DLSR and its related variants, our MLG-LSC implicitly carries out the e-dragging and can naturally reflect the roughly relative distance relationships among the categories from a metric learning perspective. Furthermore, our optimization objective functions are strictly (geodesically) convex and thus can obtain their corresponding closed-form solutions, resulting in higher computational performance. Experimental results on a set of benchmark data sets indicate the validity of our learning framework.

[1]  Chih-Jen Lin,et al.  Coordinate Descent Method for Large-scale L2-loss Linear Support Vector Machines , 2008, J. Mach. Learn. Res..

[2]  Lin Wu,et al.  Robust Subspace Clustering for Multi-View Data by Exploiting Correlation Consensus , 2015, IEEE Transactions on Image Processing.

[3]  Tilo Strutz,et al.  Data Fitting and Uncertainty: A practical introduction to weighted least squares and beyond , 2010 .

[4]  Koby Crammer,et al.  On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines , 2002, J. Mach. Learn. Res..

[5]  Feiping Nie,et al.  Discriminative Least Squares Regression for Multiclass Classification and Feature Selection , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[6]  D. Hosmer,et al.  Applied Logistic Regression , 1991 .

[7]  Suvrit Sra,et al.  Geometric Mean Metric Learning , 2016, ICML.

[8]  Bruno Iannazzo,et al.  The geometric mean of two matrices from a computational viewpoint , 2011, Numer. Linear Algebra Appl..

[9]  Zhihui Lai,et al.  Manifold discriminant regression learning for image classification , 2015, Neurocomputing.

[10]  Sameer A. Nene,et al.  Columbia Object Image Library (COIL100) , 1996 .

[11]  Lin Wu,et al.  Unsupervised Metric Fusion Over Multiview Data by Graph Random Walk-Based Cross-View Diffusion , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Lingfeng Wang,et al.  MSDLSR: Margin Scalable Discriminative Least Squares Regression for Multicategory Classification , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[13]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[14]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[15]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[16]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[17]  Stanley Lemeshow,et al.  Applied Logistic Regression, Second Edition , 1989 .

[18]  Shiming Xiang,et al.  Retargeted Least Squares Regression Algorithm , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[20]  Svetha Venkatesh,et al.  Face Recognition Using Kernel Ridge Regression , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Chunhong Pan,et al.  Groupwise Retargeted Least-Squares Regression , 2018, IEEE Transactions on Neural Networks and Learning Systems.