A K-nearest neighbours method based on imprecise probabilities

K-nearest neighbours algorithms are among the most popular existing classification methods, due to their simplicity and good performances. Over the years, several extensions of the initial method have been proposed. In this paper, we propose a K-nearest neighbours approach that uses the theory of imprecise probabilities, and more specifically lower previsions. We show that the proposed approach has several assets: it can handle uncertain data in a very generic way, and decision rules developed within this theory allow us to deal with conflicting information between neighbours or with the absence of close neighbour to the instance to classify. We show that results of the basic k-NN and weighted k-NN methods can be retrieved by the proposed approach. We end with some experiments on the classical data sets.

[1]  Asit P. Basu,et al.  Probabilistic Risk Analysis , 2002 .

[2]  Leon N. Cooper,et al.  Improving nearest neighbor rule with a simple adaptive distance measure , 2006, Pattern Recognit. Lett..

[3]  Sébastien Destercke,et al.  A Decision Rule for Imprecise Probabilities Based on Pair-Wise Comparison of Expectation Bounds , 2010, SMPS.

[4]  Arie Tzvieli.  Possibility theory: An approach to computerized processing of uncertainty , 1990, J. Am. Soc. Inf. Sci..

[5]  Luis M. de Campos,et al.  Probability Intervals: a Tool for uncertain Reasoning , 1994, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[6]  J. Jaffray,et al.  Information Processing under Imprecise Risk with the Hurwicz criterion , 2007 .

[7]  Yan Qiu Chen,et al.  Improving nearest neighbor classification with cam weighted distance , 2006, Pattern Recognit..

[8]  James O. Berger,et al.  An overview of robust Bayesian analysis , 1994 .

[9]  J. Kacprzyk,et al.  Aggregation and Fusion of Imperfect Information , 2001 .

[10]  I. Gilboa,et al.  Maxmin Expected Utility with Non-Unique Prior , 1989 .

[11]  Gert de Cooman,et al.  Extreme lower probabilities , 2008, Fuzzy Sets Syst..

[12]  James M. Keller,et al.  A fuzzy K-nearest neighbor algorithm , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[13]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[14]  Roberto Manduchi,et al.  Stereo Matching as a Nearest-Neighbor Problem , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  B. M. Hill,et al.  Theory of Probability , 1990 .

[16]  Enrique Miranda,et al.  A survey of the theory of coherent lower previsions , 2008, Int. J. Approx. Reason..

[17]  Hideyuki Imai,et al.  Probably correct k-nearest neighbor search in high dimensions , 2010, Pattern Recognit..

[18]  Thierry Denoeux,et al.  An evidence-theoretic k-NN rule with parameter optimization , 1998, IEEE Trans. Syst. Man Cybern. Part C.

[19]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[20]  Lotfi A. Zadeh,et al.  The concept of a linguistic variable and its application to approximate reasoning-III , 1975, Inf. Sci..

[21]  Giuliana Regoli,et al.  Inference under imprecise probability assessments , 1999, Soft Comput..

[22]  Chih-Fong Tsai,et al.  A triangle area based nearest neighbors approach to intrusion detection , 2010, Pattern Recognit..

[23]  Jean-Marc Bernard,et al.  An introduction to the imprecise Dirichlet model for multinomial data , 2005, Int. J. Approx. Reason..

[24]  Philippe Smets,et al.  The Transferable Belief Model , 1991, Artif. Intell..

[25]  Yu. I. Lyubich.  Foundations and Methods , 1992 .

[26]  Bernard Dubuisson,et al.  A statistical decision rule with incomplete knowledge about classes , 1993, Pattern Recognit..

[27]  Chaur-Heh Hsieh,et al.  Fast search algorithms for vector quantization of images using multiple triangle inequalities and wavelet transform , 2000, IEEE Trans. Image Process..

[28]  Roger M. Cooke,et al.  Probabilistic Risk Analysis: Probabilistic risk analysis , 2001 .

[29]  Eyke Hüllermeier,et al.  Case-Based Approximate Reasoning (Theory and Decision Library B) , 2007 .

[30]  Lotfi A. Zadeh,et al.  The Concepts of a Linguistic Variable and its Application to Approximate Reasoning , 1975 .

[31]  Serafín Moral,et al.  Aggregation of Imprecise Probabilities , 1998 .

[32]  Marco Zaffalon,et al.  Utility-Based Accuracy Measures to Empirically Evaluate Credal Classifiers , 2011 .

[33]  Marco Zaffalon,et al.  Lazy naive credal classifier , 2009, U '09.

[34]  Robert Tibshirani,et al.  Discriminant Adaptive Nearest Neighbor Classification , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[35]  Lev V. Utkin.  Extensions of belief functions and possibility distributions by using the imprecise Dirichlet model , 2005, Fuzzy Sets Syst..

[36]  Thierry Denoeux,et al.  A k-nearest neighbor classification rule based on Dempster-Shafer theory , 1995, IEEE Trans. Syst. Man Cybern..

[37]  Sébastien Destercke.  A K-Nearest Neighbours Method Based on Lower Previsions , 2010, IPMU.

[38]  Yi-Ping Hung,et al.  Fast and versatile algorithm for nearest neighbor search based on a lower bound tree , 2007, Pattern Recognit..

[39]  P. Walley.  Statistical Reasoning with Imprecise Probabilities , 1990 .

[40]  P. M. Williams,et al.  Notes on conditional previsions , 2007, Int. J. Approx. Reason..

[41]  Sahibsingh A. Dudani.  The Distance-Weighted k-Nearest-Neighbor Rule , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[42]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[43]  C. K. Chow,et al.  On optimum recognition error and reject tradeoff , 1970, IEEE Trans. Inf. Theory.

[44]  Nicu Sebe,et al.  Boosting the distance estimation: Application to the K-Nearest Neighbor Classifier , 2006, Pattern Recognit. Lett..

[45]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[46]  Grigorios Tsoumakas,et al.  Random k -Labelsets: An Ensemble Method for Multilabel Classification , 2007, ECML.

[47]  Matthias C. M. Troffaes.  Decision making under uncertainty using imprecise probabilities , 2007, Int. J. Approx. Reason..

[48]  Eyke Hüllermeier,et al.  Case-Based Approximate Reasoning , 2007, Theory and Decision Library.