Supervised discriminant Isomap with maximum margin graph regularization for dimensionality reduction

Abstract As one of the most popular nonlinear dimensionality reduction methods, Isomap has been widely used in pattern recognition and machine learning. However, Isomap has the following problems: (1) Isomap is an unsupervised dimensionality reduction method, it cannot use class label information to obtain discriminative low dimensional embedding for classification; (2) The embedding performance of Isomap is sensitive to neighborhood size parameter; (3) Isomap cannot deal with outside new data by direct embedding. In this paper, a novel dimensionality reduction method called supervised discriminant Isomap is proposed to solve the first two problems mentioned above. Specifically, first, raw data points are partitioned into different manifolds by using their class label information. Then, supervised discriminant Isomap aims at seeking an optimal nonlinear subspace to preserve the geometrical structure of each manifold according to the Isomap criterion, and to enhance the discriminating capability by maximizing the distances between data points of different classes and the maximum margin graph regularization term. Finally, the corresponding optimization problems are solved by using eigen-decomposition algorithm. Further, we extend supervised discriminant Isomap to a linear dimensionality reduction method called supervised discriminant Isomap projection for handling the above three problems. Moreover, our approaches have three important characteristics: (1) Proposed methods adaptively estimate the local neighborhood surrounding each sample based on data density and similarity; (2) The objective functions of proposed methods can maximize margins between the each classes in the dimension-reduced feature space; (3) The objective functions of proposed methods have closed-form solutions. Furthermore, our methods can capture more discriminative information from raw data than other Isomap based methods. Extensive experiments on nine data sets demonstrate that the proposed methods are superior to the related state-of-the-art methods.

[1]  Yan Zhang,et al.  Semi-supervised local multi-manifold Isomap by linear embedding for feature extraction , 2018, Pattern Recognit..

[2]  Stefan Hougardy,et al.  The Floyd-Warshall algorithm on graphs with negative cycles , 2010, Inf. Process. Lett..

[3]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[4]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[5]  Li Zhang,et al.  Joint Label Consistent Dictionary Learning and Adaptive Label Prediction for Semisupervised Machine Fault Classification , 2016, IEEE Transactions on Industrial Informatics.

[6]  Bogdan Raducanu,et al.  A supervised non-linear dimensionality reduction approach for manifold learning , 2012, Pattern Recognit..

[7]  Patrick J. F. Groenen,et al.  Modern Multidimensional Scaling: Theory and Applications , 2003 .

[8]  Kilian Q. Weinberger,et al.  An Introduction to Nonlinear Dimensionality Reduction by Maximum Variance Unfolding , 2006, AAAI.

[9]  Xiangtao Zheng,et al.  Dimensionality Reduction by Spatial–Spectral Preservation in Selected Bands , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[11]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[12]  Joshua B. Tenenbaum,et al.  Global Versus Local Methods in Nonlinear Dimensionality Reduction , 2002, NIPS.

[13]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[14]  Shekhar Verma,et al.  Optimal manifold neighborhood and kernel width for robust non-linear dimensionality reduction , 2019, Knowl. Based Syst..

[15]  Michael Biehl,et al.  A General Framework for Dimensionality-Reducing Data Visualization Mapping , 2012, Neural Computation.

[16]  Ann B. Lee,et al.  Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Zhipeng Ge,et al.  Effect of dimensionality reduction on stock selection with cluster analysis in different market situations , 2020, Expert Syst. Appl..

[18]  Jiawei Han,et al.  Isometric Projection , 2007, AAAI.

[19]  Mehrbakhsh Nilashi,et al.  A recommender system based on collaborative filtering using ontology and dimensionality reduction techniques , 2018, Expert Syst. Appl..

[20]  R. Fisher THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[21]  Li Zhang,et al.  Joint Low-Rank and Sparse Principal Feature Coding for Enhanced Robust Representation and Visual Classification , 2016, IEEE Transactions on Image Processing.

[22]  W. Scott Spangler,et al.  Class visualization of high-dimensional data with applications , 2002, Comput. Stat. Data Anal..

[23]  Muhammad Kashif Hanif,et al.  Overview and comparative study of dimensionality reduction techniques for high dimensional data , 2020, Inf. Fusion.

[24]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[25]  Ali Aghagolzadeh,et al.  Local distances preserving based manifold learning , 2020, Expert Syst. Appl..

[26]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[27]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[28]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[29]  Murat Kaya,et al.  A large electroencephalographic motor imagery dataset for electroencephalographic brain computer interfaces , 2018, Scientific Data.

[30]  Mukund Balasubramanian,et al.  The Isomap Algorithm and Topological Stability , 2002, Science.

[31]  Mansoor Zolghadri Jahromi,et al.  Supervised discriminative dimensionality reduction by learning multiple transformation operators , 2021, Expert Syst. Appl..

[32]  Nojun Kwak,et al.  Principal Component Analysis Based on L1-Norm Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Sedat Korkmaz,et al.  DEBOHID: A differential evolution based oversampling approach for highly imbalanced datasets , 2021, Expert Syst. Appl..

[34]  Tommy W. S. Chow,et al.  M-Isomap: Orthogonal Constrained Marginal Isomap for Nonlinear Dimensionality Reduction , 2013, IEEE Transactions on Cybernetics.

[35]  Zhi-Hua Zhou,et al.  Supervised nonlinear dimensionality reduction for visualization and classification , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[36]  M. Sniedovich Dynamic programming : foundations and principles , 2011 .

[37]  I. Jolliffe Principal Component Analysis and Factor Analysis , 1986 .

[38]  Bo Yang,et al.  Multi-manifold Discriminant Isomap for visualization and classification , 2016, Pattern Recognit..