Correlated Component Regression: A Prediction/Classification Methodology for Possibly Many Features

A new ensemble dimension reduction regression technique, called Correlated Component Regression (CCR), is proposed that predicts the dependent variable based on K correlated components. For K = 1, CCR is equivalent to the corresponding Naive Bayes solution, and for K = P, CCR is equivalent to traditional regression with P predictors. An optional step-down variable selection procedure provides a sparse solution, with each component defined as a linear combination of only P* < P predictors. For high-dimensional data, simulation results suggest that good prediction is generally attainable for K = 3 or 4 regardless of the number of predictors, and estimation is fast. When predictors include one or more suppressor variables, common with gene expression data, simulations based on linear regression, logistic regression and discriminant analysis suggest that CCR predicts outside the sample better than comparable approaches based on stepwise regression, penalized regression and/or PLS regression. A major reason for the improvement is that the CCR/step-down algorithm is much better than other sparse techniques in capturing important suppressor variables among the final predictors.