Predictive power of principal components for single-index model and sufficient dimension reduction

In this paper we demonstrate that a higher-ranking principal component of the predictor tends to have a stronger correlation with the response in single index models and sufficient dimension reduction. This tendency holds even though the orientation of the predictor is not designed in any way to be related to the response. This provides a probabilistic explanation of why it is often beneficial to perform regression on principal components—a practice commonly known as principal component regression but whose validity has long been debated. This result is a generalization of earlier results by Li (2007) [19], Artemiou and Li (2009) [2], and Ni (2011) [24], where the same phenomenon was conjectured and rigorously demonstrated for linear regression.

[2]  L. Ferré,et al.  Functional sliced inverse regression analysis , 2003 .

[3]  Thomas M. Stoker,et al.  Semiparametric Estimation of Index Coefficients , 1989 .

[4]  R. Cook,et al.  Dimension reduction for conditional mean in regression , 2002 .

[5]  Barry C. Arnold,et al.  On distributions whose component ratios are Cauchy , 1992 .

[6]  Bing Li,et al.  A general theory for nonlinear sufficient dimension reduction: Formulation and estimation , 2013, 1304.0580.

[7]  Ker-Chau Li,et al.  Sliced Inverse Regression for Dimension Reduction , 1991 .

[8]  Bing Li,et al.  Determining the dimension of iterative Hessian transformation , 2004 .

[9]  Michael I. Jordan,et al.  Kernel dimension reduction in regression , 2009, 0908.1854.

[10]  H. Ichimura,et al.  SEMIPARAMETRIC LEAST SQUARES (SLS) AND WEIGHTED SLS ESTIMATION OF SINGLE-INDEX MODELS , 1993 .

[11]  Su-Yun Huang,et al.  Nonlinear Dimension Reduction with Kernel Sliced Inverse Regression , 2009, IEEE Transactions on Knowledge and Data Engineering.

[12]  Bing Li,et al.  Principal support vector machines for linear and nonlinear sufficient dimension reduction , 2011, 1203.2790.

[13]  W. Härdle,et al.  Optimal Smoothing in Single-index Models , 1993 .

[14]  Liqiang Niu.  PRINCIPAL COMPONENT REGRESSION REVISITED , 2011 .

[15]  Shaoli Wang,et al.  On Directional Regression for Dimension Reduction , 2007 .

[16]  R. H. Moore,et al.  Regression Graphics: Ideas for Studying Regressions Through Graphics , 1998, Technometrics.

[17]  Peter Hall,et al.  Ordering and selecting components in multivariate or functional data linear prediction , 2010 .

[18]  T. Hsing,et al.  An RKHS formulation of the inverse regression dimension-reduction problem , 2009, 0904.0076.

[19]  R. F. Ling,et al.  Some cautionary notes on the use of principal components regression , 1998 .

[20]  N. Altman,et al.  On dimension folding of matrix- or array-valued statistical objects , 2010, 1002.4789.

[21]  Bing Li.  Comment: Fisher Lecture: Dimension Reduction in Regression , 2007, 0708.3777.

[22]  H. Tong,et al.  Article: 2 , 2002, European Financial Services Law.

[23]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[24]  F. Chiaromonte,et al.  Dimension reduction strategies for analyzing global gene expression data with a response. , 2002, Mathematical biosciences.

[25]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[26]  R. Christensen,et al.  Fisher Lecture: Dimension Reduction in Regression , 2007, 0708.3774.

[27]  Bing Li,et al.  ON PRINCIPAL COMPONENTS AND REGRESSION: A STATISTICAL EXPLANATION OF A NATURAL PHENOMENON , 2009 .

[28]  I. Jolliffe.  A Note on the Use of Principal Components in Regression , 1982 .

[29]  Han-Ming Wu.  Kernel Sliced Inverse Regression with Applications to Classification , 2008 .

[30]  R. Cook.  Graphics for regressions with a binary response , 1996 .