A Comparison of Variable Selection Methods with the Main Focus on Orthogonalization

When multidimensional data has to be approximated, linear regression models are widely used. If the number of input space dimensions is very high, dimension reduction methods have to be applied for several reasons. Variable selection (in the literature also called subset selection) is a special kind of dimension reduction as it tries to find the smallest set of regressors that approximates a given independent best. Several methods for selecting these regressor variables exist, including forward selection, backward elimination and stepwise regression — see [7] and [8]. In this paper, we assume that the number of regressor candidates is very high, and the number of variables in the finally used linear regression model is quite low. Under these circumstances, several methods cannot be used because they result in a too high computational effort. Therefore, variants of forward selection are preferred because they are the fastest variable selection methods. Especially two variants of forward selection with orthogonalization are investigated, because orthogonalization improves the quality of the result.