New Routes from Minimal Approximation Error to Principal Components

We introduce two new methods of deriving the classical PCA in the framework of minimizing the mean square error upon performing a lower-dimensional approximation of the data. These methods are based on two forms of the mean square error function. One of the novelties of the presented methods is that the commonly employed process of subtraction of the mean of the data becomes part of the solution of the optimization problem and not a pre-analysis heuristic. We also derive the optimal basis and the minimum error of approximation in this framework and demonstrate the elegance of our solution in comparison with a recent solution in the framework.

[1]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[2]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[3]  Don J. Torrieri,et al.  Mine detection using scattering parameters , 1997, IEEE Trans. Neural Networks.

[4]  Sabine Van Huffel,et al.  Recent advances in total least squares techniques and errors-in-variables modeling , 1997 .

[5]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[6]  David G. Stork,et al.  Pattern Classification , 1973 .

[7]  A. Mahalanobis,et al.  Design and application of quadratic correlation filters for target detection , 2004, IEEE Transactions on Aerospace and Electronic Systems.

[8]  Chein-I Chang,et al.  Hyperspectral image classification and dimensionality reduction: an orthogonal subspace projection approach , 1994, IEEE Trans. Geosci. Remote. Sens..

[9]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[10]  N. L. Johnson,et al.  Multivariate Analysis , 1958, Nature.

[11]  James V. Candy,et al.  Adaptive and Learning Systems for Signal Processing, Communications, and Control , 2006 .

[12]  Yoshua Bengio,et al.  Pattern Recognition and Neural Networks , 1995 .

[13]  Charles E. Heckler,et al.  Applied Multivariate Statistical Analysis , 2005, Technometrics.

[14]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[15]  Keinosuke Fukunaga,et al.  Application of the Karhunen-Loève Expansion to Feature Selection and Ordering , 1970, IEEE Trans. Computers.

[16]  Don Torrieri,et al.  Mine Detection Using Scattering Parameters and an Artificial Neural Network , 1997 .

[17]  Malcolm K. Hughes,et al.  corrigendum: Global-scale temperature patterns and climate forcing over the past six centuries , 2004 .

[18]  I. Noy-Meir,et al.  Data Transformations in Ecological Ordination: I. Some Advantages of Non-Centering , 1973 .

[19]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .

[20]  Paul F. Whelan,et al.  Fukunaga-koontz transform for small sample size problems , 2005 .

[21]  Ross McKitrick,et al.  Reply to comment by Huybers on “Hockey sticks, principal components, and spurious significance” , 2005 .

[22]  W. T. Williams,et al.  Data Transformations in Ecological Ordination: II. On the Meaning of Data Standardization , 1975 .

[23]  Michael Elad,et al.  Optimal reduced-rank quadratic classifiers using the Fukunaga-Koontz transform with applications to automated target recognition , 2003, SPIE Defense + Commercial Sensing.

[24]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[25]  Sun-Yuan Kung,et al.  Principal Component Neural Networks: Theory and Applications , 1996 .

[26]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[27]  Juha Karhunen,et al.  Principal component neural networks — Theory and applications , 1998, Pattern Analysis and Applications.

[28]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[29]  Malcolm K. Hughes,et al.  Global-scale temperature patterns and climate forcing over the past six centuries , 1998, Nature.