Deep Learning in current Neuroimaging: a multivariate approach with power and type I error control but arguable generalization ability

Discriminative analysis in neuroimaging by means of deep/machine learning techniques is usually tested with validation techniques such as K-fold, hold-out, and resubstitution, whereas the associated statistical significance remains largely under-developed due to their computational complexcarmenj@ugr.es javierrp@ugr.es js369@cam.ac.uk Jonathan.Voeglein@med.uni-muenchen.de Johannes.Levin@med.uni-muenchen.de gorriz@ugr.es, jg528@cam.ac.uk Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf Deep Learning in current Neuroimaging: a multivariate approach with power and type I error control but arguable generalization ability A PREPRINT ity. In this work, a non-parametric framework is proposed that estimates the statistical significance of classifications using deep learning architectures that are increasingly used in neuroimaging. In particular, a combination of autoencoders (AE) and support vector machines (SVM) is applied to: (i) a one-condition, within-group designs often of normal controls (NC) and; (ii) a two-condition, between-group designs which contrast, for example, Alzheimer’s disease (AD) patients with NC (the extension to multi-class analyses is also included). A random-effects inference based on a label permutation test is proposed in both studies using cross-validation (CV) and resubstitution with upper bound correction (RUB) as validation methods. This allows both false positives and classifier overfitting to be detected as well as estimating the statistical power of the test. Several experiments were carried out using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset, the Dominantly Inherited Alzheimer Network (DIAN) dataset, and a MCI prediction dataset. We found in the permutation test that CV and RUB methods offer a false positive rate close to the significance level and an acceptable statistical power (although lower using cross-validation). A large separation between training and test accuracies using CV was observed, especially in one-condition designs. This implies a low generalization ability as the model fitted in training is not informative with respect to the test set. We propose as solution by applying RUB, whereby similar results are obtained to those of the CV test set, but considering the whole set and with a lower computational cost per iteration.