Regularization and improved interpretation of linear data mappings and adaptive distance measures

Linear data transformations are essential operations in many machine learning algorithms, helping to make such models more flexible or to emphasize certain data directions. In particular for high dimensional data sets linear transformations are not necessarily uniquely determined, though, and alternative parameterizations exist which do not change the mapping of the training data. Thus, regularization is required to make the model robust to noise and more interpretable for the user. In this contribution, we characterize the group of transformations which leave a linear mapping invariant for a given finite data set, and we discuss the consequences on the interpretability of the models. We propose an intuitive regularization mechanism to avoid problems in under-determined configurations, and we test the approach in two machine learning models.

[1]  Barbara Hammer,et al.  Relevance learning in generative topographic mapping , 2011, Neurocomputing.

[2]  Michael Biehl,et al.  Distance Learning in Discriminative Vector Quantization , 2009, Neural Computation.

[3]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[4]  Barbara Hammer,et al.  Relational Generative Topographic Map , 2010, ESANN.

[5]  Michael Seifert,et al.  Posterior regularization and attribute assessment of under-determined linear mappings , 2012, ESANN.

[6]  Constantin F. Aliferis,et al.  A theoretical characterization of linear SVM-based feature selection , 2004, ICML '04.

[7]  Thomas Villmann,et al.  Generalized relevance learning vector quantization , 2002, Neural Networks.

[8]  Ju Wang,et al.  PCA-based SVM for automatic recognition of gait patterns. , 2008, Journal of applied biomechanics.

[9]  Michael Biehl,et al.  Matrix relevance LVQ in steroid metabolomics based classification of adrenal tumors , 2012, ESANN.

[10]  Eric O. Postma,et al.  Dimensionality Reduction: A Comparative Review , 2008 .

[11]  Terrence J. Sejnowski,et al.  Slow Feature Analysis: Unsupervised Learning of Invariances , 2002, Neural Computation.

[12]  Thomas Villmann,et al.  Regularization in Matrix Relevance Learning , 2010, IEEE Transactions on Neural Networks.

[13]  Stefan Rüping,et al.  Learning interpretable models , 2006 .

[14]  Michel Verleysen,et al.  Nonlinear Dimensionality Reduction , 2021, Computer Vision.

[15]  Barbara Hammer,et al.  Local matrix adaptation in topographic neural maps , 2011, Neurocomputing.

[16]  Michael Biehl,et al.  Adaptive Relevance Matrices in Learning Vector Quantization , 2009, Neural Computation.

[17]  Matthew O. Ward,et al.  Interactive Data Visualization: Foundations, Techniques, and Applications, Second Edition - 360 Degree Business , 2015 .

[18]  Thomas Villmann,et al.  Large margin linear discriminative visualization by Matrix Relevance Learning , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[19]  Michel Verleysen,et al.  Feature clustering and mutual information for the selection of variables in spectral data , 2007, ESANN.

[20]  Kilian Q. Weinberger,et al.  Convex Optimizations for Distance Metric Learning and Pattern Classification [Applications Corner] , 2010, IEEE Signal Processing Magazine.

[21]  Michael Biehl,et al.  Urine Steroid Metabolomics as a Biomarker Tool for Detecting Malignancy in Adrenal Tumors , 2011, The Journal of clinical endocrinology and metabolism.

[22]  Ignacio Ponzoni,et al.  Multi‐Objective Feature Selection in QSAR Using a Machine Learning Approach , 2009 .

[23]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[24]  Jieping Ye,et al.  Adaptive Distance Metric Learning for Clustering , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Matthew O. Ward,et al.  Interactive Data Visualization - Foundations, Techniques, and Applications , 2010 .

[26]  Thomas Villmann,et al.  Limited Rank Matrix Learning, discriminative dimension reduction and visualization , 2012, Neural Networks.

[27]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[28]  Paulo J. G. Lisboa,et al.  Making machine learning models interpretable , 2012, ESANN.

[29]  Erkki Oja,et al.  Independent component analysis: algorithms and applications , 2000, Neural Networks.

[30]  Rong Jin,et al.  Regularized Distance Metric Learning: Theory and Algorithm , 2009, NIPS.