Regression on manifolds: Estimation of the exterior derivative

Collinearity and near-collinearity of predictors cause difficulties when doing regression. In these cases, variable selection becomes untenable because of mathematical issues concerning the existence and numerical stability of the regression coefficients, and interpretation of the coefficients is ambiguous because gradients are not defined. Using a differential geometric interpretation, in which the regression coefficients are interpreted as estimates of the exterior derivative of a function, we develop a new method to do regression in the presence of collinearities. Our regularization scheme can improve estimation error, and it can be easily modified to include lasso-type regularization. These estimators also have simple extensions to the "large $p$, small $n$" context.

[1]  W. Massy.  Principal Components Regression in Exploratory Statistical Research , 1965 .

[2]  R. R. Hocking.  The analysis and selection of variables in linear regression , 1976 .

[3]  D. Freedman,et al.  Some Asymptotic Theory for the Bootstrap , 1981 .

[4]  Generalized inverse matrices and their applications , 1982 .

[5]  D. Sattinger,et al.  Calculus on Manifolds , 1986 .

[6]  I. Helland.  ON THE STRUCTURE OF PARTIAL LEAST SQUARES REGRESSION , 1988 .

[7]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[8]  Sabine Van Huffel,et al.  Total least squares problem - computational aspects and analysis , 1991, Frontiers in applied mathematics.

[9]  J. Friedman,et al.  A Statistical View of Some Chemometrics Regression Tools , 1993 .

[10]  J. Friedman,et al.  [A Statistical View of Some Chemometrics Regression Tools]: Response , 1993 .

[11]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[12]  Jianqing Fan,et al.  Nonparametric regression with errors in variables , 1993 .

[13]  M. Wand,et al.  Multivariate Locally Weighted Least Squares Regression , 1994 .

[14]  C. Geyer.  On the Asymptotics of Constrained $M$-Estimation , 1994 .

[15]  Jianqing Fan,et al.  Local polynomial modelling and its applications , 1994 .

[16]  Jian-Hua Shao,et al.  Bootstrap Sample Size in Nonregular Cases , 1994 .

[17]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[18]  J. Shao.  Bootstrap Model Selection , 1996 .

[19]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[20]  Andrew Y. Ng,et al.  Preventing "Overfitting" of Cross-Validation Data , 1997, ICML.

[21]  G. Lugosi,et al.  On Concentration-of-Measure Inequalities , 1998 .

[22]  D. Ruppert,et al.  Nonparametric regression in the presence of measurement error , 1999 .

[23]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[24]  Wenjiang J. Fu.  Ridge estimator in singulah oesiun with application to age-period-cohort analysis of disease rates , 2000 .

[25]  Wenjiang J. Fu,et al.  Asymptotics for lasso-type estimators , 2000 .

[26]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[27]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[28]  John M. Lee.  Introduction to Smooth Manifolds , 2002 .

[29]  Juha Reunanen,et al.  Overfitting in Making Comparisons Between Variable Selection Methods , 2003, J. Mach. Learn. Res..

[30]  Alfred O. Hero,et al.  Geodesic entropic graphs for dimension and entropy estimation in manifold learning , 2004, IEEE Transactions on Signal Processing.

[31]  Wenjiang J. Fu,et al.  2. A Methodological Comparison of Age-Period-Cohort Models: The Intrinsic Estimator and Conventional Generalized Linear Models , 2004 .

[32]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[33]  Peter J. Bickel,et al.  Maximum Likelihood Estimation of Intrinsic Dimension , 2004, NIPS.

[34]  Matthias Hein.  Intrinsic Dimensionality Estimation of Submanifolds in R , 2005 .

[35]  Larry A. Wasserman,et al.  Rodeo: Sparse Nonparametric Regression in High Dimensions , 2005, NIPS.

[36]  Matthias Hein,et al.  Intrinsic dimensionality estimation of submanifolds in Rd , 2005, ICML.

[37]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[38]  R. Tibshirani,et al.  Sparse Principal Component Analysis , 2006 .

[39]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[40]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[41]  Terence Tao,et al.  The Dantzig selector: Statistical estimation when P is much larger than n , 2005, math/0506081.

[42]  R. Bhatia.  Perturbation Bounds for Matrix Eigenvalues , 2007 .

[43]  Regina Y. Liu,et al.  Complex datasets and inverse problems : tomography, networks and beyond , 2007, 0708.1130.

[44]  R. Tibshirani,et al.  PATHWISE COORDINATE OPTIMIZATION , 2007, 0708.1485.

[45]  P. Bickel,et al.  Local polynomial regression on unknown manifolds , 2007, 0708.0983.

[46]  Noureddine El Karoui,et al.  Operator norm consistent estimation of large-dimensional sparse covariance matrices , 2008, 0901.3220.

[47]  Alon Zakai,et al.  Manifold Learning: The Price of Normalization , 2008, J. Mach. Learn. Res..

[48]  Glenn Fung,et al.  On the Dangers of Cross-Validation. An Experimental Evaluation , 2008, SDM.

[49]  N. Meinshausen,et al.  Stability selection , 2008, 0809.2932.

[50]  B. Nadler,et al.  Determining the number of components in a factor model from limited noisy data , 2008 .

[51]  K. Lange,et al.  Coordinate descent algorithms for lasso penalized regression , 2008, 0803.3876.

[52]  Wenjiang J. Fu.  A Smoothing Cohort Model in Age–Period–Cohort Analysis With Applications to Homicide Arrest Rates and Lung Cancer Mortality Rates , 2008 .

[53]  N. Meinshausen,et al.  LASSO-TYPE RECOVERY OF SPARSE REPRESENTATIONS FOR HIGH-DIMENSIONAL DATA , 2008, 0806.0145.

[54]  I. Johnstone,et al.  Sparse Principal Components Analysis , 2009, 0901.4392.

[55]  I. Johnstone,et al.  On Consistency and Sparsity for Principal Components Analysis in High Dimensions , 2009, Journal of the American Statistical Association.

[56]  B. Nadler.  Finite sample approximation results for principal component analysis: a matrix perturbation approach , 2009, 0901.3245.

[57]  Claire J. Tomlin,et al.  Statistics for sparse, high-dimensional, and nonparametric system identification , 2009, 2009 IEEE International Conference on Robotics and Automation.

[58]  Allen Y. Yang,et al.  Robust Face Recognition via Sparse Representation , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Martin Andersson,et al.  A comparison of nine PLS1 algorithms , 2009 .

[60]  P. Bickel,et al.  Covariance regularization by thresholding , 2009, 0901.3079.

[61]  James Brown,et al.  Nonparametric identification of regulatory interactions from spatial and temporal gene expression data , 2010, BMC Bioinformatics.

[62]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[63]  Partha Niyogi,et al.  Manifold regularization and semi-supervised learning: some theoretical analyses , 2013, J. Mach. Learn. Res..