Local feature selection for heterogeneous problems

Current electronic data repositories contain enormous amount of data including also unknown and potentially interesting patterns and relations. One approach commonly used is supervised machine learning, in which a set of training instances is used to train one or more classifiers that map the space formed by different features of the instances into the set of class values. The classifiers are later used to classify new instances with unknown class values. The multidimensional data is sometimes feature-space heterogeneous so that different features have different importance in different subareas of the whole space. In this paper we describe a technique that searches for a division of the feature space identifying the best subsets of features for each instance. Our technique is based on the wrapper approach where a classification algorithm is used as the evaluation function to differentiate between several feature subsets. On the other hand, to make the feature selection dynamic or local, we apply the recently developed technique for dynamic integration of classifiers to determine what classifier and with what feature subset is applied for each new instance. Our technique can be applied in the case of implicit heterogeneity when the regions of heterogeneity cannot be easily defined by a simple dependency. We make experiments with well-known datasets of the UCI machine learning repository using ensembles of simple base classifiers. The base classifiers discussed in the experiments are developed by C4.5 based on only one feature each but the combining method itself is allowed to take into account unlimited amount of present features. The results achieved are promising and show that the local data mining in comparison with mining the whole space can be advantageous. In many cases, the mining accuracy is better as well as the time of processing is shorter, because only a small set of features is used to classify each new instance.