MSc Project Feature Selection using Information Theoretic Techniques

This document presents a investigation into 3 different areas of feature selection, using information theoretic methods. The first area of research is an investigation into the selection of the first feature in common feature selection algorithms. This step is often overlooked in the construction of feature selection algorithms, with the assumption that the most informative feature is the best first choice. This can be proven to be untrue, and so an investigation into how to select a better suited feature forms the first part of the research. New methods for selecting the first feature are proposed and empirically tested to see if they offer an improvement over the standard method. The second area of research is an investigation into applying the Renyi extension to information theory to standard feature selection techniques. This requires the development of a Renyi mutual information measure, and two different measures are proposed. The Renyi extension provides a positive real parameter, α, which can be varied. The new Renyi feature selection techniques are empirically tested, varying the measure and value of α used. The third area of research is an investigation into a different method of estimating the Renyi entropy of a variable, by using a function based upon the length of the minimal spanning tree of the variable. This enables a high dimensional estimate of the entropy to be constructed in O(s log s) time and can be used to implement the Max Dependency criterion. This research investigates the use of this estimator as a feature selection criterion and proposes a new genetic algorithm based method for selecting features, in contrast to the traditional forward search used in other algorithms.