Permutation distributions of fMRI classification do not behave in accord with central limit theorem

Applying Monte Carlo method on Fisher's exact test is a prominent choice in estimating an existent statistical effect in large data. When used to analyze classification results, the method, which is widely known as permutation testing, works by testing the null hypothesis after generating a permutation distribution (PD) of classification accuracies/errors that is centered around chance-level. In principle, these PDs should behave in accord with the central limit theorem (CLT) if the independence condition in the cross-validation classification error (test statistic) is fulfilled. Permutation testing has been widely used in pattern classification applied to neuroimaging studies to eradicate chance performance. In this work, we used Anderson-Darling test to evaluate the accordance level of PDs of classification accuracies to normality expected under CLT. An exhaustive simulation study was carried out using functional magnetic resonance imaging data that were collected while human subjects responded to visual stimulation paradigms and the following classifiers were considered: support vector machines, logistic regression, ridge logistic regression, Gaussian Naive Bayes, sparse multinomial logistic regression, and artificial neural networks. Our results showed that while the standard normal distribution does not adequately fit to PDs, it tends to fit well when the mean classification accuracy averaged over a set of independent classifiers is considered. We also found that across-run lk motion correction of the fMRI data weakens the accordance of PDs with CLT and this phenomenon could be due to the across runs dependence resulting from motion correction.