The permutation test for feature selection by mutual information

The estimation of mutual information for feature selection is often subject to inaccuracies due to noise, small sample size, bad choice of parameter for the estimator, etc. The choice of a threshold above which a feature will be considered useful is thus difficult to make. Therefore, the use of the permutation test to assess the reliability of the estimation is proposed. The permutation test allows performing a non-parametric hypothesis test to select the relevant features and to build a Feature Relevance Diagram that visually synthesizes the result of the test.