Intrinsic Dimension Estimation by Maximum Likelihood in Probabilistic PCA

A central issue in dimension reduction is choosing a sensible number of dimensions to be retained. This work demonstrates the asymptotic consistency of the maximum likelihood criterion for determining the intrinsic dimension of a dataset in a isotropic version of Probabilistic Principal Component Analysis (PPCA). Numerical experiments on simulated and real datasets show that the maximum likelihood criterion can actually be used in practice and outperforms existing intrinsic dimension selection criteria in various situations. This paper exhibits as well the limits of the maximum likelihood criterion and recommends in specific situations the use of the AIC criterion.

[1]  K. I. WilliamsDivision,et al.  Products of Gaussians and Probabilistic Minor Component Analysis , 2002, Neural Computation.

[2]  Guillaume Bouchard,et al.  Selection of generative models in classification , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  C. Schmid,et al.  High-Dimensional Discriminant Analysis , 2005 .

[4]  Keinosuke Fukunaga,et al.  An Algorithm for Finding Intrinsic Dimensionality of Data , 1971, IEEE Transactions on Computers.

[5]  Tom Minka,et al.  Automatic Choice of Dimensionality for PCA , 2000, NIPS.

[6]  A. Vinciarelli,et al.  Estimating the Intrinsic Dimension of Data with a Fractal-Based Method , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[8]  Francesco Camastra,et al.  Data dimensionality estimation methods: a survey , 2003, Pattern Recognit..

[9]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[10]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[11]  Bernard Chalmond,et al.  Nonlinear Modeling of Scattered Multivariate Data and Its Application to Shape Change , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Bo Zhang,et al.  Intrinsic dimension estimation of manifolds by incising balls , 2009, Pattern Recognit..

[13]  Peter J. Bickel,et al.  Maximum Likelihood Estimation of Intrinsic Dimension , 2004, NIPS.

[14]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[15]  Balázs Kégl,et al.  Intrinsic Dimension Estimation Using Packing Numbers , 2002, NIPS.

[16]  H. Akaike.  A new look at the statistical model identification , 1974 .

[17]  Juha Karhunen,et al.  Representation and separation of signals using nonlinear PCA type learning , 1994, Neural Networks.

[18]  Max Welling,et al.  Extreme Components Analysis , 2003, NIPS.

[19]  Alexander Basilevsky,et al.  Statistical Factor Analysis and Related Methods , 1994 .

[20]  Gerald Sommer,et al.  Intrinsic Dimensionality Estimation With Optimally Topology Preserving Maps , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  David E. Tyler.  Asymptotic Inference for Eigenvectors , 1981 .

[22]  Robert P. W. Duin,et al.  An Evaluation of Intrinsic Dimensionality Estimators , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Christopher M. Bishop,et al.  Mixtures of Probabilistic Principal Component Analyzers , 1999, Neural Computation.

[24]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[25]  Anil K. Jain,et al.  An Intrinsic Dimensionality Estimator from Near-Neighbor Information , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Stan Lipovetsky,et al.  Latent Variable Models and Factor Analysis , 2001, Technometrics.

[27]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[28]  R. Cattell.  The Scree Test For The Number Of Factors. , 1966, Multivariate behavioral research.