Interpretable Locally Adaptive Nearest Neighbors

Abstract When training automated systems, it has been shown to be beneficial to adapt the representation of data by learning a problem-specific metric. This metric is global. We extend this idea and, for the widely used family of k nearest neighbors algorithms, develop a method that allows learning locally adaptive metrics. These local metrics not only improve performance, but are naturally interpretable. To demonstrate important aspects of how our approach works, we conduct a number of experiments on synthetic data sets, and we show its usefulness on real-world benchmark data sets.

[1]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[2]  Kilian Q. Weinberger,et al.  Fast solvers and efficient implementations for distance metric learning , 2008, ICML '08.

[3]  Kevin Baker,et al.  Classification of radar returns from the ionosphere using neural networks , 1989 .

[4]  Marc Sebban,et al.  A Survey on Metric Learning for Feature Vectors and Structured Data , 2013, ArXiv.

[5]  Byoung-Tak Zhang,et al.  Generative Local Metric Learning for Nearest Neighbor Classification , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Klaus-Robert Müller,et al.  iNNvestigate neural networks! , 2018, J. Mach. Learn. Res..

[7]  Fevzi Alimo.  Methods of Combining Multiple Classiiers Based on Diierent Representations for Pen-based Handwritten Digit Recognition , 1996 .

[8]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[9]  Ethem Alpaydin,et al.  Cascading classifiers , 1998, Kybernetika.

[10]  J. Knapp,et al.  CORSIKA: A Monte Carlo code to simulate extensive air showers , 1998 .

[11]  M. V. Velzen,et al.  Self-organizing maps , 2007 .

[12]  Thomas Villmann,et al.  Generalized relevance learning vector quantization , 2002, Neural Networks.

[13]  Nicolai Petkov,et al.  Marker selection for the detection of trisomy 21 using generalized matrix learning vector quantization , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[14]  Heiko Wersing,et al.  Self-Adjusting Memory: How to Deal with Diverse Drift Types , 2017, IJCAI.

[15]  Barbara Hammer,et al.  Locally Adaptive Nearest Neighbors , 2020, ESANN.

[16]  Barbara Hammer,et al.  Interpretation of linear classifiers by means of feature relevance bounds , 2018, Neurocomputing.

[17]  Michael Biehl,et al.  Adaptive Relevance Matrices in Learning Vector Quantization , 2009, Neural Computation.

[18]  Michael Biehl,et al.  Matrix relevance LVQ in steroid metabolomics based classification of adrenal tumors , 2012, ESANN.

[19]  Alexandros Kalousis,et al.  Parametric Local Metric Learning for Nearest Neighbor Classification , 2012, NIPS.

[20]  Carlos Guestrin,et al.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier , 2016, ArXiv.

[21]  Jonathan J. Hull,et al.  A Database for Handwritten Text Recognition Research , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[23]  Seth Flaxman,et al.  European Union Regulations on Algorithmic Decision-Making and a "Right to Explanation" , 2016, AI Mag..

[24]  David J. Slate,et al.  Letter Recognition Using Holland-Style Adaptive Classifiers , 1991, Machine Learning.

[25]  Guilherme A. Barreto,et al.  Short-term memory mechanisms in neural network learning of robot navigation tasks: A case study , 2009, 2009 6th Latin American Robotics Symposium (LARS 2009).

[26]  Heiko Wersing,et al.  KNN Classifier with Self Adjusting Memory for Heterogeneous Concept Drift , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[27]  Leland McInnes,et al.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction , 2018, ArXiv.

[28]  Frank-Michael Schleif,et al.  On the Generalization Ability of Prototype-Based Classifiers with Local Relevance Determination , 2005 .

[29]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[30]  John W. Sammon,et al.  A Nonlinear Mapping for Data Structure Analysis , 1969, IEEE Transactions on Computers.

[31]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[32]  Heiko Wersing,et al.  Mitigating Concept Drift via Rejection , 2018, ICANN.