A review of unsupervised feature selection methods

In recent years, unsupervised feature selection methods have raised considerable interest in many research areas; this is mainly due to their ability to identify and select relevant features without needing class label information. In this paper, we provide a comprehensive and structured review of the most relevant and recent unsupervised feature selection methods reported in the literature. We present a taxonomy of these methods and describe the main characteristics and the fundamental ideas they are based on. Additionally, we summarized the advantages and disadvantages of the general lines in which we have categorized the methods analyzed in this review. Moreover, an experimental comparison among the most representative methods of each approach is also presented. Finally, we discuss some important open challenges in this research area.

[1]  Ferat Sahin,et al.  A survey on feature selection methods , 2014, Comput. Electr. Eng..

[2]  Lei Wang,et al.  The Effect of the Characteristics of the Dataset on the Selection Stability , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[3]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[4]  Yiu-ming Cheung,et al.  Feature Selection and Kernel Learning for Local Learning-Based Clustering , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Steve Mansfield-Devine,et al.  Data classification: keeping track of your most precious asset , 2016, Netw. Secur..

[6]  Jun Guo,et al.  Dependence Guided Unsupervised Feature Selection , 2018, AAAI.

[7]  Lei Shi,et al.  Robust Spectral Learning for Unsupervised Feature Selection , 2014, 2014 IEEE International Conference on Data Mining.

[8]  Raúl Santos-Rodríguez,et al.  Spectral Clustering and Feature Selection for Microarray Data , 2009, 2009 International Conference on Machine Learning and Applications.

[9]  Huan Liu,et al.  An Unsupervised Feature Selection Framework for Social Media Data , 2014, IEEE Transactions on Knowledge and Data Engineering.

[10]  M. R. Osborne,et al.  On the LASSO and its Dual , 2000 .

[11]  M. Punithavalli,et al.  Survey on Feature Selection in Document Clustering , 2011 .

[12]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[13]  Manoranjan Dash,et al.  Dimensionality reduction of unsupervised data , 1997, Proceedings Ninth IEEE International Conference on Tools with Artificial Intelligence.

[14]  Jaya Sil,et al.  Simultaneous feature selection and clustering with mixed features by multi objective genetic algorithm , 2014, Int. J. Hybrid Intell. Syst..

[15]  Feiping Nie,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Feature Selection via Joint Embedding Learning and Sparse Regression , 2022 .

[16]  Douglas H. Fisher,et al.  Knowledge Acquisition Via Incremental Conceptual Clustering , 1987, Machine Learning.

[17]  ChengXiang Zhai,et al.  Robust Unsupervised Feature Selection , 2013, IJCAI.

[18]  Lei Wang,et al.  Efficient Spectral Feature Selection with Minimum Redundancy , 2010, AAAI.

[19]  Massimiliano Pontil,et al.  Convex multi-task feature learning , 2008, Machine Learning.

[20]  Habibollah Haron,et al.  Supervised, Unsupervised, and Semi-Supervised Feature Selection: A Review on Gene Selection , 2016, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[21]  Jian Zhang,et al.  Unsupervised spectral feature selection with l1-norm graph , 2016, Neurocomputing.

[22]  Pablo A. Estévez,et al.  A review of feature selection methods based on mutual information , 2013, Neural Computing and Applications.

[23]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[24]  M. Phil,et al.  Survey on Feature Selection in Document Clustering , 2011 .

[25]  Hiroshi Motoda,et al.  Computational Methods of Feature Selection , 2022 .

[26]  David A. McAllester,et al.  Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , 2009, UAI 2009.

[27]  Mário A. T. Figueiredo,et al.  An unsupervised approach to feature discretization and selection , 2012, Pattern Recognit..

[28]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[29]  Yaakov Tsaig,et al.  Fast Solution of $\ell _{1}$ -Norm Minimization Problems When the Solution May Be Sparse , 2008, IEEE Transactions on Information Theory.

[30]  C. A. Murthy,et al.  Unsupervised Feature Selection Using Feature Similarity , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Sam Lightstone,et al.  Data Mining - Know It All , 2008 .

[32]  Xuelong Li,et al.  Unsupervised Feature Selection with Structured Graph Optimization , 2016, AAAI.

[33]  Muhammad Sharif,et al.  Intelligent Image Retrieval Techniques: A Survey , 2014 .

[34]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[35]  Jianbo Yu,et al.  A hybrid feature selection scheme and self-organizing map model for machine health assessment , 2011, Appl. Soft Comput..

[36]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[37]  Witold Pedrycz,et al.  Unsupervised feature selection via maximum projection and minimum redundancy , 2015, Knowl. Based Syst..

[38]  M. Friedman.  The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[39]  Deng Cai,et al.  Unsupervised feature selection for multi-cluster data , 2010, KDD.

[40]  Huan Liu,et al.  Toward integrating feature selection algorithms for classification and clustering , 2005, IEEE Transactions on Knowledge and Data Engineering.

[41]  Huan Liu,et al.  Spectral feature selection for supervised and unsupervised learning , 2007, ICML '07.

[42]  Yun Li,et al.  Hierarchical fuzzy filter method for unsupervised feature selection , 2007, J. Intell. Fuzzy Syst..

[43]  Josef Kittler,et al.  Pattern recognition : a statistical approach , 1982 .

[44]  Anil K. Jain,et al.  Unsupervised Learning of Finite Mixture Models , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Shikha Agrawal,et al.  Survey on Anomaly Detection using Data Mining Techniques , 2015, KES.

[46]  Pedro Larrañaga,et al.  A review of feature selection techniques in bioinformatics , 2007, Bioinform..

[47]  Anil K. Jain,et al.  Simultaneous feature selection and clustering using mixture models , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[49]  Salvatore J. Stolfo,et al.  Adaptive Intrusion Detection: A Data Mining Approach , 2000, Artificial Intelligence Review.

[50]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[51]  Jianyu Miao,et al.  A Survey on Feature Selection , 2016 .

[52]  Jianzhong Wang,et al.  Unsupervised Feature Selection with Graph Regularized Nonnegative Self-representation , 2016, CCBR.

[53]  Yong Shi,et al.  Feature Selection With $\ell_{2,1-2}$ Regularization. , 2018, IEEE transactions on neural networks and learning systems.

[54]  Jing Liu,et al.  Clustering-Guided Sparse Structural Learning for Unsupervised Feature Selection , 2014, IEEE Transactions on Knowledge and Data Engineering.

[55]  Kezhi Mao,et al.  Identifying critical variables of principal components for unsupervised feature selection , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[56]  Jing Liu,et al.  Unsupervised Feature Selection Using Nonnegative Spectral Analysis , 2012, AAAI.

[57]  Reza Zafarani,et al.  Social Media Mining: An Introduction , 2014 .

[58]  Jesús Ariel Carrasco-Ochoa,et al.  A new Unsupervised Spectral Feature Selection Method for mixed data: A filter approach , 2017, Pattern Recognit..

[59]  Simon C. K. Shiu,et al.  Unsupervised feature selection by regularized self-representation , 2015, Pattern Recognit..

[60]  Han Wang,et al.  Unsupervised feature selection via low-rank approximation and structure learning , 2017, Knowl. Based Syst..

[61]  Huan Liu,et al.  Spectral Feature Selection for Data Mining , 2011 .

[62]  Qinghua Zheng,et al.  Adaptive Unsupervised Feature Selection With Structure Regularization , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[63]  Mohammad Ali Zare Chahooki,et al.  A Survey on semi-supervised feature selection methods , 2017, Pattern Recognit..

[64]  D. Botstein,et al.  Singular value decomposition for genome-wide expression data processing and modeling. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[65]  Zi Huang,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence ℓ2,1-Norm Regularized Discriminative Feature Selection for Unsupervised Learning , 2022 .

[66]  Filippo Menczer,et al.  Evolutionary model selection in unsupervised learning , 2002, Intell. Data Anal..

[67]  Chris H. Q. Ding,et al.  Robust nonnegative matrix factorization using L21-norm , 2011, CIKM '11.

[68]  Rongcheng Liu,et al.  An Unsupervised Feature Selection Algorithm: Laplacian Score Combined with Distance-Based Entropy Measure , 2009, 2009 Third International Symposium on Intelligent Information Technology Application.

[69]  Manoranjan Dash,et al.  RELIEF-C: Efficient Feature Selection for Clustering over Noisy Data , 2011, 2011 IEEE 23rd International Conference on Tools with Artificial Intelligence.

[70]  Zhuwen Li,et al.  SCAMS: Simultaneous Clustering and Model Selection , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[71]  Bernhard Schölkopf,et al.  A Local Learning Approach for Clustering , 2006, NIPS.

[72]  S. Sitharama Iyengar,et al.  Data-Driven Techniques in Disaster Information Management , 2017, ACM Comput. Surv..

[73]  Jay Lee,et al.  A hybrid feature selection scheme for unsupervised learning and its application in bearing fault diagnosis , 2011, Expert Syst. Appl..

[74]  Pichao Wang,et al.  Robust unsupervised feature selection via dual self-representation and manifold regularization , 2018, Knowl. Based Syst..

[75]  M. E. Maron,et al.  Automatic Indexing: An Experimental Inquiry , 1961, JACM.

[76]  Huan Liu,et al.  Feature selection for clustering - a filter solution , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[77]  Liang-Chieh Chen,et al.  Unsupervised Feature Selection: Minimize Information Redundancy of Features , 2010, 2010 International Conference on Technologies and Applications of Artificial Intelligence.

[78]  Nikhil R. Pal,et al.  Feature selection with SVD entropy: Some modification and extension , 2014, Inf. Sci..

[79]  David Zhang,et al.  Non-convex Regularized Self-representation for Unsupervised Feature Selection , 2015, IScIDE.

[80]  Deng Cai,et al.  Laplacian Score for Feature Selection , 2005, NIPS.

[81]  Jing Wang,et al.  Swarm Intelligence in Cellular Robotic Systems , 1993 .

[82]  G. Ritter.  Robust Cluster Analysis and Variable Selection , 2014 .

[83]  Juyang Weng,et al.  Efficient content-based image retrieval using automatic feature selection , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[84]  Pramod Kumar Singh,et al.  A Survey on Filter Techniques for Feature Selection in Text Mining , 2012, SocProS.

[85]  Hugues Bersini,et al.  A Survey on Filter Techniques for Feature Selection in Gene Expression Microarray Analysis , 2012, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[86]  Xiangjian He,et al.  Unsupervised Feature Selection Method for Intrusion Detection System , 2015, 2015 IEEE Trustcom/BigDataSE/ISPA.

[87]  E. Fowlkes,et al.  Variable selection in clustering , 1988 .

[88]  J. F. Chin,et al.  Feature selection in multimedia: The state-of-the-art review , 2017, Image Vis. Comput..

[89]  Michal Linial,et al.  Novel Unsupervised Feature Filtering of Biological Data , 2006, ISMB.

[90]  Parham Moradi,et al.  Gene selection for microarray data classification using a novel ant colony optimization , 2015, Neurocomputing.

[91]  Mark A. Hall,et al.  Correlation-based Feature Selection for Machine Learning , 2003 .

[92]  Ali Zakerolhosseini,et al.  Unsupervised probabilistic feature selection using ant colony optimization , 2016, Expert Syst. Appl..

[93]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[94]  Huan Liu,et al.  Feature Selection for Clustering: A Review , 2018, Data Clustering: Algorithms and Applications.

[95]  Hongwei Hao,et al.  Selecting feature subset with sparsity and low redundancy for unsupervised learning , 2015, Knowl. Based Syst..

[96]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[97]  Zhexue Huang,et al.  CLUSTERING LARGE DATA SETS WITH MIXED NUMERIC AND CATEGORICAL VALUES , 1997 .

[98]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[99]  Joshua Zhexue Huang,et al.  Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values , 1998, Data Mining and Knowledge Discovery.

[100]  S. Niijima,et al.  Laplacian Linear Discriminant Analysis Approach to Unsupervised Feature Selection , 2009, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[101]  Salem Alelyani,et al.  On Feature Selection Stability: A Data Perspective , 2013 .

[102]  Xing Zhou,et al.  An Improved Text Clustering Method based on Hybrid Model , 2009 .

[103]  Verónica Bolón-Canedo,et al.  Feature selection for high-dimensional data , 2016, Progress in Artificial Intelligence.

[104]  Ron Kohavi,et al.  Feature Selection for Knowledge Discovery and Data Mining , 1998 .

[105]  Sankar K. Pal,et al.  Unsupervised feature evaluation: a neuro-fuzzy approach , 2000, IEEE Trans. Neural Networks Learn. Syst..

[106]  Md. Rafiqul Islam,et al.  A survey of anomaly detection techniques in financial domain , 2016, Future Gener. Comput. Syst..

[107]  Kewei Cheng,et al.  Feature Selection , 2016, ACM Comput. Surv..

[108]  George Forman,et al.  An Extensive Empirical Study of Feature Selection Metrics for Text Classification , 2003, J. Mach. Learn. Res..

[109]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[110]  Huan Liu,et al.  Spectral feature selection for mining ultrahigh dimensional data , 2010 .

[111]  Yoh-Han Pao,et al.  Statistical Pattern Recognition. Second edition (Keinosuke Fukunaga) , 1993, SIAM Rev..

[112]  Parham Moradi,et al.  An unsupervised feature selection algorithm based on ant colony optimization , 2014, Eng. Appl. Artif. Intell..

[113]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[114]  Francisco Herrera,et al.  Data Preprocessing in Data Mining , 2014, Intelligent Systems Reference Library.

[115]  Julius T. Tou,et al.  Pattern Recognition Principles , 1974 .

[116]  Manoranjan Dash,et al.  Feature Selection for Clustering , 2009, Encyclopedia of Database Systems.

[117]  Andrew R. Webb,et al.  Statistical Pattern Recognition , 1999 .

[118]  Beatriz de la Iglesia,et al.  Survey on Feature Selection , 2015, ArXiv.

[119]  Huan Liu,et al.  Feature Engineering for Machine Learning and Data Analytics , 2018 .

[120]  Young Bun Kim,et al.  Unsupervised Gene Selection For High Dimensional Data , 2006, Sixth IEEE Symposium on BioInformatics and BioEngineering (BIBE'06).

[121]  Parham Moradi,et al.  Relevance-redundancy feature selection based on ant colony optimization , 2015, Pattern Recognit..

[122]  Haitao Liu,et al.  A hybrid feature selection scheme for mixed attributes data , 2013 .

[123]  Henri Luchian,et al.  A unifying criterion for unsupervised clustering and feature selection , 2011, Pattern Recognit..

[124]  Ali A. Ghorbani,et al.  An Iterative Hybrid Filter-Wrapper Approach to Feature Selection for Document Clustering , 2009, Canadian Conference on AI.

[125]  K. P. Singh,et al.  Support vector machines in water quality management. , 2011, Analytica chimica acta.

[126]  John C. Davis,et al.  Book Review: Introduction to statistical pattern recognition. 2nd edition, by Keinosuke Fukunaga, Academic Press, San Diego, 1990, 591 p., ISBN 0-12-269851-7, US$69.95 , 1996 .

[127]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[128]  Constantine Kotropoulos,et al.  Feature Selection Based on Mutual Correlation , 2006, CIARP.

[129]  Estevam R. Hruschka,et al.  Feature selection for clustering problems: a hybrid algorithm that iterates between k-means and a Bayesian filter , 2005, Fifth International Conference on Hybrid Intelligent Systems (HIS'05).

[130]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[131]  Xuelong Li,et al.  Joint Embedding Learning and Sparse Regression: A Framework for Unsupervised Feature Selection , 2014, IEEE Transactions on Cybernetics.

[132]  DuttaDipankar,et al.  Simultaneous feature selection and clustering with mixed features by multi objective genetic algorithm , 2014 .

[133]  V. N. Sastry,et al.  Unsupervised feature ranking based on representation entropy , 2012, 2012 1st International Conference on Recent Advances in Information Technology (RAIT).

[134]  Jinhui Tang,et al.  Unsupervised Feature Selection via Nonnegative Spectral Analysis and Redundancy Control , 2015, IEEE Transactions on Image Processing.

[135]  T. Caliński,et al.  A dendrite method for cluster analysis , 1974 .

[136]  Huan Liu,et al.  Feature Selection for Classification: A Review , 2014, Data Classification: Algorithms and Applications.

[137]  Volker Roth,et al.  Feature Selection in Clustering Problems , 2003, NIPS.

[138]  Carla E. Brodley,et al.  Feature Selection for Unsupervised Learning , 2004, J. Mach. Learn. Res..

[139]  Xuelong Li,et al.  Structure preserving unsupervised feature selection , 2018, Neurocomputing.

[140]  Michael J Daniels,et al.  Longitudinal profiling of health care units based on continuous and discrete patient outcomes. , 2005, Biostatistics.

[141]  Yihui Luo,et al.  Clustering Ensemble for Unsupervised Feature Selection , 2009, 2009 Sixth International Conference on Fuzzy Systems and Knowledge Discovery.

[142]  Yun Li,et al.  A Hybrid Method of Unsupervised Feature Selection Based on Ranking , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[143]  Eduardo R. Hruschka,et al.  Feature Selection for Cluster Analysis: an Approach Based on the Simplified Silhouette Criterion , 2005, International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC'06).

[144]  C. Lursinsap,et al.  Univariate Filter Technique for Unsupervised Feature Selection Using a New Laplacian Score Based Local Nearest Neighbors , 2009, 2009 Asia-Pacific Conference on Information Processing.

[145]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[146]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[147]  Jie Tian,et al.  Robust graph regularized unsupervised feature selection , 2018, Expert Syst. Appl..

[148]  K. Thangavel,et al.  Unsupervised adaptive floating search feature selection based on Contribution Entropy , 2010, 2010 International Conference on Communication and Computational Intelligence (INCOCCI).

[149]  Ashwin Ram,et al.  Efficient Feature Selection in Conceptual Clustering , 1997, ICML.

[150]  Jieping Ye,et al.  Multi-Task Feature Learning Via Efficient l2, 1-Norm Minimization , 2009, UAI.

[151]  Jesús Ariel Carrasco-Ochoa,et al.  A new hybrid filter-wrapper feature selection method for clustering based on ranking , 2016, Neurocomputing.

[152]  Lei Wang,et al.  On Similarity Preserving Feature Selection , 2013, IEEE Transactions on Knowledge and Data Engineering.

[153]  Alexander R. De Leon,et al.  Analysis of Mixed Data : Methods & Applications , 2013 .

[154]  Guoliang Luo,et al.  Structure Preserving Non-negative Feature Self-Representation for Unsupervised Feature Selection , 2017, IEEE Access.

[155]  Qi Mao,et al.  Feature selection for unsupervised learning through local learning , 2015, Pattern Recognit. Lett..

[156]  S B Kotsiantis,et al.  RETRACTED ARTICLE: Feature selection for machine learning classification problems: a recent overview , 2014, Artificial Intelligence Review.

[157]  Yide Ma,et al.  Robust unsupervised feature selection via matrix factorization , 2017, Neurocomputing.

[158]  Shulin Wang,et al.  Feature selection in machine learning: A new perspective , 2018, Neurocomputing.

[159]  Luis Talavera,et al.  Dependency-based feature selection for clustering symbolic data , 2000, Intell. Data Anal..

[160]  LarrañagaPedro,et al.  A review of feature selection techniques in bioinformatics , 2007 .

[161]  Nassima Dif,et al.  Gene Selection for Microarray Data Classification Using Hybrid Meta-Heuristics , 2018, MISC.

[162]  Luca Maria Gambardella,et al.  Ant colony system: a cooperative learning approach to the traveling salesman problem , 1997, IEEE Trans. Evol. Comput..