Supervised feature selection through Deep Neural Networks with pairwise connected structure

Abstract Feature selection is an important data preprocessing strategy, has been proven empirically that it contributes to reducing the dimensionality of feature and enhancing the performance of learning algorithms in practice. Typical sparse learning-based models select the features by removing ones that the feature scores are zero. However, linear models puzzle to build the non-linear relations between features and responses. The Deep Neural Network (DNN) has a strong capability to mode the non-linear relations and has been employed to select features. In this paper, we introduce a novel deep Neural network-based Feature Selection (NeuralFS) method to identify features. The new model comprises of a fully-connection network, a decision network, and connect them through a pairwise connected structure. In NeuralFS, the fully-connected network is the crucial structure in NeuralFS that transforms the features into their corresponding scores, and the decision network is the final structure that performs classification or regression. The pairwise connected can be regarded as a “bridge” to connect the two networks, and its weights are fixed as the normalized input as well as it is un-trainable during model training. After optimizing, the feature scores can be obtained by calculating the output of the fully-connected network. NeuralFS takes advantage of the deep network to model the non-linearity, and also make features scores sparse without the sparse regularization technology. We apply the proposed method to both synthetic datasets and benchmark datasets to prove its effectiveness.