Principal component analysis - a tutorial

Dimensionality reduction is one of the preprocessing steps in many machine learning applications and it is used to transform the features into a lower dimension space. Principal Component Analysis (PCA) technique is one of the most famous unsupervised dimensionality reduction techniques. The goal of the PCA is to find the space, which represents the direction of the maximum variance of the given data. This paper highlights the basic background needed to understand and implement the PCA technique. This paper starts with basic definitions of the PCA technique and the algorithms of two methods of calculating PCA, namely, the covariance matrix and Singular Value Decomposition (SVD) methods. Moreover, a number of numerical examples are illustrated to show how the PCA space is calculated in easy steps. Three experiments are conducted to show how to apply PCA in the real applications including biometrics, image compression, and visualization of high-dimensional datasets.

[1]  V. Kshirsagar,et al.  Face recognition using Eigenfaces , 2011, 2011 3rd International Conference on Computer Research and Development.

[2]  Gilbert Strang,et al.  Introduction to applied mathematics , 1988 .

[3]  Xihong Lin,et al.  Sparse linear discriminant analysis for simultaneous testing for the significance of a gene set/pathway and gene selection , 2009, Bioinform..

[4]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[5]  Merle C. Potter,et al.  Differential Equations with Linear Algebra , 2009 .

[6]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[7]  Richard A. Harshman,et al.  Noise Reduction in BOLD-Based fMRI Using Component Analysis , 2002, NeuroImage.

[8]  Václav Snásel,et al.  Plant Identification: Two Dimensional-Based Vs. One Dimensional-Based Feature Extraction Methods , 2015, SOCO.

[9]  Jonathon Shlens,et al.  A Tutorial on Principal Component Analysis , 2014, ArXiv.

[10]  D. Botstein,et al.  Singular value decomposition for genome-wide expression data processing and modeling. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Pedro Larrañaga,et al.  A review of feature selection techniques in bioinformatics , 2007, Bioinform..

[12]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[13]  M. A. Carreira-Perpinan,et al.  Compression neural networks for feature extraction: Application to human recognition from ear images , 1995 .

[14]  Gerald Schaefer,et al.  Ear Recognition Using Block-Based Principal Component Analysis and Decision Fusion , 2015, PReMI.

[15]  A. Tharwat,et al.  Personal identification using ear images based on fast and accurate principal component analysis , 2012, 2012 8th International Conference on Informatics and Systems (INFOS).

[16]  Heidrun Schumann,et al.  Enhancing the visualization process with principal component analysis to support the exploration of trends , 2006, APVIS.

[17]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[18]  Leo H. Chiang,et al.  Fault diagnosis in chemical processes using Fisher discriminant analysis, discriminant partial least squares, and principal component analysis , 2000 .

[19]  Daniel T. Larose,et al.  Discovering Knowledge in Data: An Introduction to Data Mining , 2005 .

[20]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[21]  Heikki Mannila,et al.  Principles of Data Mining , 2001, Undergraduate Topics in Computer Science.

[22]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[23]  Jian Yang,et al.  Two-dimensional PCA: a new approach to appearance-based face representation and recognition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Jin-Rong Cui.  Multispectral palmprint recognition using Image-Based Linear Discriminant Analysis , 2012, Int. J. Biom..

[25]  Jarkko Venna,et al.  Information Retrieval Perspective to Nonlinear Dimensionality Reduction for Data Visualization , 2010, J. Mach. Learn. Res..

[26]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[27]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[28]  Nanda Kambhatla,et al.  Dimension Reduction by Local Principal Component Analysis , 1997, Neural Computation.

[29]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[30]  Yachen Lin,et al.  Geometric Data Analysis: An Empirical Approach to Dimensionality Reduction and the Study of Patterns , 2002, Technometrics.

[31]  Manoranjan Dash,et al.  Dimensionality reduction of unsupervised data , 1997, Proceedings Ninth IEEE International Conference on Tools with Artificial Intelligence.

[32]  R. Tibshirani,et al.  Discriminant Analysis by Gaussian Mixtures , 1996 .

[33]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[34]  Zohreh Azimifar,et al.  Supervised principal component analysis: Visualization, classification and regression on subspaces and submanifolds , 2011, Pattern Recognit..

[35]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.