Toward value difference metric with attribute weighting

In distance metric learning, recent work has shown that value difference metric (VDM) with a strong attribute independence assumption outperforms other existing distance metrics. However, an open question is whether VDM with a less restrictive assumption can perform even better. Many approaches have been proposed to improve VDM by weakening the assumption. In this paper, we make a comprehensive survey on the existing improved approaches and then propose a new approach to improve VDM by attribute weighting. We name the proposed new distance function as attribute-weighted value difference metric (AWVDM). Moreover, we propose a modified attribute-weighted value difference metric (MAWVDM) by incorporating the learned attribute weights into the conditional probability estimates of AWVDM. AWVDM and MAWVDM significantly outperform VDM and inherit the computational simplicity of VDM simultaneously. Experimental results on a large number of UCI data sets validate the performance of AWVDM and MAWVDM.

[1]  Rudolf Fleischer,et al.  Distance Approximating Dimension Reduction of Riemannian Manifolds , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[2]  Meng Wang,et al.  Semisupervised Multiview Distance Metric Learning for Cartoon Synthesis , 2012, IEEE Transactions on Image Processing.

[3]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[4]  David W. Aha,et al.  A Probabilistic Framework for Memory-Based Reasoning , 1998, Artif. Intell..

[5]  Enver Sangineto,et al.  Pose and Expression Independent Facial Landmark Localization Using Dense-SURF and the Hausdorff Distance , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Liangxiao Jiang,et al.  A Novel Distance Function: frequency difference Metric , 2014, Int. J. Pattern Recognit. Artif. Intell..

[7]  S. Salzberg,et al.  A weighted nearest neighbor algorithm for learning with symbolic features , 2004, Machine Learning.

[8]  Hongwei Li,et al.  Local value difference metric , 2014, Pattern Recognit. Lett..

[9]  Bernhard Pfahringer,et al.  Locally Weighted Naive Bayes , 2002, UAI.

[10]  Russell Greiner,et al.  Discriminative Model Selection for Belief Net Structures , 2005, AAAI.

[11]  Harry Zhang,et al.  Learning weighted naive Bayes with accurate ranking , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[12]  F. Oppacher,et al.  Evolutionary Data Mining With Automatic Rule Generalization , 2001 .

[13]  Jesús Alcalá-Fdez,et al.  KEEL Data-Mining Software Tool: Data Set Repository, Integration of Algorithms and Experimental Analysis Framework , 2011, J. Multiple Valued Log. Soft Comput..

[14]  David W. Aha,et al.  Tolerating Noisy, Irrelevant and Novel Attributes in Instance-Based Learning Algorithms , 1992, Int. J. Man Mach. Stud..

[15]  Keinosuke Fukunaga,et al.  The optimal distance measure for nearest neighbor classification , 1981, IEEE Trans. Inf. Theory.

[16]  Hongwei Li,et al.  Naive Bayes for value difference metric , 2014, Frontiers of Computer Science.

[17]  Shasha Wang,et al.  Attribute Weighted Value Difference Metric , 2013, 2013 IEEE 25th International Conference on Tools with Artificial Intelligence.

[18]  Rong Jin,et al.  Distance Metric Learning: A Comprehensive Survey , 2006 .

[19]  Liangxiao Jiang,et al.  An Augmented Value Difference Measure , 2013, Pattern Recognit. Lett..

[20]  Yuan Yan Tang,et al.  High-Order Distance-Based Multiview Stochastic Learning in Image Classification , 2014, IEEE Transactions on Cybernetics.

[21]  Yi Yang,et al.  Learning a 3D Human Pose Distance Metric from Geometric Pose Descriptor , 2011, IEEE Transactions on Visualization and Computer Graphics.

[22]  Tony R. Martinez,et al.  Improved Heterogeneous Distance Functions , 1996, J. Artif. Intell. Res..

[23]  Hongwei Li,et al.  Selective Value Difference Metric , 2013, J. Comput..

[24]  David J. Hand,et al.  The multi-class metric problem in nearest neighbour discrimination rules , 1990, Pattern Recognit..

[25]  Liangxiao Jiang,et al.  Not always simple classification: Learning SuperParent for class probability estimation , 2015, Expert Syst. Appl..

[26]  Jun Yu,et al.  Semantic preserving distance metric learning and applications , 2014, Inf. Sci..

[27]  Meng Wang,et al.  Joint Learning of Labels and Distance Metric , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[28]  Victor Cheng,et al.  Dissimilarity learning for nominal data , 2004, Pattern Recognit..

[29]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[30]  Hongwei Li,et al.  A Modified Short and Fukunaga Metric based on the attribute independence assumption , 2012, Pattern Recognit. Lett..

[31]  Shasha Wang,et al.  Deep feature weighting for naive Bayes and its application to text classification , 2016, Eng. Appl. Artif. Intell..

[32]  Mark A. Hall,et al.  A decision tree-based attribute weighting filter for naive Bayes , 2006, Knowl. Based Syst..

[33]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[34]  John G. Cleary,et al.  K*: An Instance-based Learner Using and Entropic Distance Measure , 1995, ICML.

[35]  Hongwei Li,et al.  One Dependence Value Difference Metric , 2011, Knowl. Based Syst..

[36]  Francesco Ricci,et al.  Probability Based Metrics for Nearest Neighbor Classification and Case-Based Reasoning , 1999, ICCBR.

[37]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[38]  Liangxiao Jiang,et al.  Naive Bayes text classifiers: a locally weighted learning approach , 2013, J. Exp. Theor. Artif. Intell..

[39]  Byoung-Tak Zhang,et al.  Generative Local Metric Learning for Nearest Neighbor Classification , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  LIANGXIAO JIANG,et al.  Discriminatively Weighted Naive Bayes and its Application in Text Classification , 2012, Int. J. Artif. Intell. Tools.

[41]  Dacheng Tao,et al.  Constrained Empirical Risk Minimization Framework for Distance Metric Learning , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Andrew W. Moore,et al.  Locally Weighted Learning , 1997, Artificial Intelligence Review.

[43]  Liangxiao Jiang,et al.  Learning Naive Bayes for Probability Estimation by Feature Selection , 2006, Canadian Conference on AI.

[44]  Yoshua Bengio,et al.  Inference for the Generalization Error , 1999, Machine Learning.

[45]  Pedro M. Domingos,et al.  Learning Bayesian network classifiers by maximizing conditional likelihood , 2004, ICML.

[46]  Michele Risi,et al.  Sketched symbol recognition using Latent-Dynamic Conditional Random Fields and distance-based clustering , 2014, Pattern Recognit..

[47]  Dacheng Tao,et al.  Person Re-Identification Over Camera Networks Using Multi-Task Distance Metric Learning , 2014, IEEE Transactions on Image Processing.

[48]  Mark A. Hall,et al.  Correlation-based Feature Selection for Discrete and Numeric Class Machine Learning , 1999, ICML.

[49]  Ian Witten,et al.  Data Mining , 2000 .

[50]  Geoffrey I. Webb,et al.  Alleviating naive Bayes attribute independence assumption by attribute weighting , 2013, J. Mach. Learn. Res..

[51]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.