Semi-supervised dimensionality reduction of hyperspectral imagery using pseudo-labels

Abstract Dimensionality reduction has been proven to be efficient in preparing high dimensional data for various tasks in machine learning. As supervised dimensionality reduction methods such as Fisher discriminant analysis (FDA) and local Fisher discriminant analysis (LFDA) tend to suffer from overfitting when only a small number of labeled samples are available, the abundant unlabeled samples could be helpful in finding a better embedding space. However, applying discriminant analysis on unlabeled data is challenging since we do not have labels for unlabeled data. In this paper, we propose a semi-supervised Semi-Supervised Local Fisher Discriminant Analysis (SSLFDA) using pseudo labels, aiming to perform discriminant analysis on both labeled and unlabeled samples. SSLFDA makes use of pseudo labels, learned from the Dirichlet process mixture model (DPMM) based clustering algorithm, to enable local Fisher discriminant analysis on unlabeled data. In addition, a kernel extension of SSLFDA is derived for non-linear dimensionality reduction. We present experimental results with real hyperspectral data to show that our method provides better classification performance compared to other existing dimensionality reduction methods.

[1]  Anil K. Jain.  Data clustering: 50 years beyond K-means , 2010, Pattern Recognit. Lett..

[2]  Huan Liu,et al.  Discriminant Analysis for Unsupervised Feature Selection , 2014, SDM.

[3]  C. Antoniak.  Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[4]  Hao Wu,et al.  Infinite Gaussian mixture models for robust decision fusion of hyperspectral imagery and full waveform LiDAR data , 2013, 2013 IEEE Global Conference on Signal and Information Processing.

[5]  Shinichi Nakajima,et al.  Semi-supervised local Fisher discriminant analysis for dimensionality reduction , 2009, Machine Learning.

[6]  Jiawei Han,et al.  Semi-supervised Discriminant Analysis , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[7]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[8]  Feiping Nie,et al.  A unified framework for semi-supervised dimensionality reduction , 2008, Pattern Recognit..

[9]  Paolo Gamba,et al.  A collection of data for urban area characterization , 2004, IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium.

[10]  Qi Wang,et al.  Dual-Clustering-Based Hyperspectral Band Selection by Contextual Analysis , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[11]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[12]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[13]  Aleksandra Pizurica,et al.  Semisupervised Local Discriminant Analysis for Feature Extraction in Hyperspectral Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[14]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[15]  Saurabh Prasad,et al.  Dirichlet Process Based Active Learning and Discovery of Unknown Classes for Hyperspectral Image Classification , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Radford M. Neal.  Probabilistic Inference Using Markov Chain Monte Carlo Methods , 2011 .

[17]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[18]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[19]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[20]  Max Welling,et al.  Accelerated Variational Dirichlet Process Mixtures , 2006, NIPS.

[21]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[22]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[23]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[24]  I. Jolliffe.  Principal Component Analysis , 2002 .

[25]  Saurabh Prasad,et al.  Morphologically Decoupled Structured Sparsity for Rotation-Invariant Hyperspectral Image Analysis , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[26]  Masashi Sugiyama,et al.  Dimensionality Reduction of Multimodal Labeled Data by Local Fisher Discriminant Analysis , 2007, J. Mach. Learn. Res..

[27]  David G. Stork,et al.  Pattern Classification , 1973 .

[28]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[29]  Feiping Nie,et al.  Semi-supervised orthogonal discriminant analysis via label propagation , 2009, Pattern Recognit..

[30]  Carl E. Rasmussen,et al.  The Infinite Gaussian Mixture Model , 1999, NIPS.

[31]  Michael J. Black,et al.  A Non-Parametric Bayesian Approach to Spike Sorting , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[32]  Radford M. Neal.  Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[33]  Qi Wang,et al.  Salient Band Selection for Hyperspectral Image Classification via Manifold Ranking , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[34]  Jing-Yu Yang,et al.  Semi-supervised linear discriminant analysis for dimension reduction and classification , 2016, Pattern Recognit..

[35]  Jing Liu,et al.  Unsupervised Feature Selection Using Nonnegative Spectral Analysis , 2012, AAAI.

[36]  Tommy W. S. Chow,et al.  A general soft label based Linear Discriminant Analysis for semi-supervised dimensionality reduction , 2014, Neural Networks.

[37]  Ying Huang,et al.  Semi-supervised Locality Preserving Discriminant Analysis for hyperspectral classification , 2016, 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[38]  ChengXiang Zhai,et al.  Robust Unsupervised Feature Selection , 2013, IJCAI.

[39]  Joachim M. Buhmann,et al.  Nonparametric Bayesian Image Segmentation , 2008, International Journal of Computer Vision.