A general theory for nonlinear sufficient dimension reduction: Formulation and estimation

In this paper we introduce a general theory for nonlinear sufficient dimension reduction, and explore its ramifications and scope. This theory subsumes recent work employing reproducing kernel Hilbert spaces, and reveals many parallels between linear and nonlinear sufficient dimension reduction. Using these parallels we analyze the properties of existing methods and develop new ones. We begin by characterizing dimension reduction at the general level of $\sigma$-fields and proceed to that of classes of functions, leading to the notions of sufficient, complete and central dimension reduction classes. We show that, when it exists, the complete and sufficient class coincides with the central class, and can be unbiasedly and exhaustively estimated by a generalized sliced inverse regression estimator (GSIR). When completeness does not hold, this estimator captures only part of the central class. However, in these cases we show that a generalized sliced average variance estimator (GSAVE) can capture a larger portion of the class. Both estimators require no numerical optimization because they can be computed by spectral decomposition of linear operators. Finally, we compare our estimators with existing methods by simulation and on actual data sets.

[1]  H. Zha,et al.  Contour regression: A general approach to dimension reduction , 2005, math/0508277.

[2]  Nonlinear Dimension Reduction , 2007 .

[3]  R. Cook,et al.  Identifying Regression Outliers and Mixtures Graphically , 2000 .

[4]  R. Christensen,et al.  Fisher Lecture: Dimension Reduction in Regression , 2007, 0708.3774.

[5]  Michael I. Jordan,et al.  Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces , 2004, J. Mach. Learn. Res..

[6]  Erich L. Lehmann,et al.  An Interpretation of Completeness and Basu's Theorem , 1981 .

[7]  Bing Li,et al.  Principal support vector machines for linear and nonlinear sufficient dimension reduction , 2011, 1203.2790.

[8]  WU Qiang,et al.  Regularized sliced inverse regression for kernel models , 2022 .

[9]  T. Hsing,et al.  An RKHS formulation of the inverse regression dimension-reduction problem , 2009, 0904.0076.

[10]  Shaoli Wang,et al.  On Directional Regression for Dimension Reduction , 2007 .

[11]  R. R. Bahadur.  Sufficiency and Statistical Decision Functions , 1954 .

[12]  Michael I. Jordan,et al.  Kernel dimension reduction in regression , 2009, 0908.1854.

[13]  Han-Ming Wu.  Kernel Sliced Inverse Regression with Applications to Classification , 2008 .

[14]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[15]  R. H. Moore,et al.  Regression Graphics: Ideas for Studying Regressions Through Graphics , 1998, Technometrics.

[16]  Bing Li,et al.  ENVELOPE MODELS FOR PARSIMONIOUS AND EFFICIENT MULTIVARIATE LINEAR REGRESSION , 2010 .

[17]  Ker-Chau Li,et al.  Regression Analysis Under Link Violation , 1989 .

[18]  Shotaro Akaho,et al.  A kernel method for canonical correlation analysis , 2006, ArXiv.

[19]  Bing Li,et al.  Successive direction extraction for estimating the central subspace in a multiple-index regression , 2008 .

[20]  R. Dennis Cook,et al.  Using Dimension-Reduction Subspaces to Identify Important Inputs in Models of Physical Systems ∗ , 2009 .

[21]  R. Cook,et al.  Likelihood-Based Sufficient Dimension Reduction , 2009 .

[22]  Su-Yun Huang,et al.  Nonlinear Dimension Reduction with Kernel Sliced Inverse Regression , 2009, IEEE Transactions on Knowledge and Data Engineering.

[23]  Ker-Chau Li,et al.  Sliced Inverse Regression for Dimension Reduction , 1991 .

[24]  Heng Tao Shen,et al.  Dimensionality Reduction , 2009, Encyclopedia of Database Systems.

[25]  Ker-Chau Li,et al.  On Principal Hessian Directions for Data Visualization and Dimension Reduction: Another Application of Stein's Lemma , 1992 .

[26]  Michael I. Jordan,et al.  Kernel independent component analysis , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[27]  R. Cook,et al.  Dimension reduction for conditional mean in regression , 2002 .

[28]  L. Ferré,et al.  Functional sliced inverse regression analysis , 2003 .

[29]  Lexin Li,et al.  Biological pathway selection through nonlinear dimension reduction. , 2011, Biostatistics.

[30]  R. Cook,et al.  Principal Hessian Directions Revisited , 1998 .

[31]  Kenji Fukumizu,et al.  Statistical Consistency of Kernel Canonical Correlation Analysis , 2007 .

[32]  R. Weiss,et al.  Using the Bootstrap to Select One of a New Class of Dimension Reduction Methods , 2003 .

[33]  C. Baker.  Joint measures and cross-covariance operators , 1973 .

[34]  R. Cook,et al.  Sufficient Dimension Reduction via Inverse Regression , 2005 .

[35]  S. Weisberg,et al.  Comments on "Sliced inverse regression for dimension reduction" by K. C. Li , 1991 .

[36]  Hongyu Zhao,et al.  Sparse Estimation of Conditional Graphical Models With Application to Gene Networks , 2012, Journal of the American Statistical Association.

[37]  W. Härdle,et al.  Optimal Smoothing in Single-index Models , 1993 .

[38]  A Bias Bound for Least Squares Linear Regression , 1991 .