Unsupervised Extraction and Supervised Selection of Features Based on Information Gain

For robust recognition we first extract features from sensory data without considering the class labels, and then select important features for the classification. The unsupervised feature extraction may incorporate Principle Component Analysis, Independent Component Analysis, and Non-negative Matrix factorization. For the supervised selection of features we adopt Fisher Score and Information Gain (IG). To avoid the calculation of multivariate joint probability density functions, instead of the IG, we use Mutual Information (MI) between a feature and the class variable. However, in this case the MI among selected features reduces the effectiveness of the feature selection, and the statistically-independent ICA-based features result in the best performance.