A RELIEF Based Feature Extraction Algorithm

RELIEF is considered one of the most successful algorithms for assessing the quality of features due to its simplicity and effectiveness. It has been recently proved that RELIEF is an online algorithm that solves a convex optimization problem with a marginbased objective function. Starting from this mathematical interpretation, we propose a novel feature extraction algorithm, referred to as LFE, as a natural generalization of RELIEF. LFE collects discriminant information through local learning, and is solved as an eigenvalue decomposition problem with a closed-form solution. A fast implementation is also derived. Experiments on synthetic and real-world data are presented. The results demonstrate that LFE performs significantly better than other feature extraction algorithms in terms of both computational efficiency and accuracy.

[1]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[2]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[3]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[4]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[5]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[6]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[7]  Enrico Tronci.  1997 , 1997, Les 25 ans de l’OMC: Une rétrospective en photos.

[8]  Jian Li,et al.  Iterative RELIEF for feature weighting , 2006, ICML.

[9]  Naftali Tishby,et al.  Margin based feature selection - theory and algorithms , 2004, ICML.

[10]  O. Nelles,et al.  An Introduction to Optimization , 1996, IEEE Antennas and Propagation Magazine.

[11]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[12]  Geoffrey E. Hinton,et al.  Neighbourhood Components Analysis , 2004, NIPS.

[13]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[14]  Thomas G. Dietterich.  Machine-Learning Research Four Current Directions , 1997 .

[15]  A. Martínez,et al.  The AR face databasae , 1998 .

[16]  Josef Kittler,et al.  Floating search methods in feature selection , 1994, Pattern Recognit. Lett..