An Industrial-Grade Brain Imaging-Based Deep Learning Classifier

Abstract Beyond detecting brain damage or tumors with magnetic resonance brain imaging, little success has been attained on identifying individual differences, e.g., sex or brain disorders. The current study aims to build an industrial-grade brain imaging-based classifier to infer individual differences using deep learning/transfer learning on big data. We pooled 34 datasets to constitute the largest brain magnetic resonance image sample to date (85,721 samples from 50,876 participants), and then applied a state-of-the-art deep convolutional neural network, Inception-ResNet-V2, to build an industrial-grade sex classifier. We achieved 94.9% accuracy in cross-dataset-validation, i.e., the model can classify the sex of a participant with brain structural imaging data from anybody and any scanner with about 95% accuracy. We then explored the potential of a deep convolutional network to objectively diagnose brain disorders. Using transfer learning, the model fine-tuned to Alzheimer’s Disease (AD) achieved 88.4% accuracy in leave-sites-out five-fold cross-validation on the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset and 86.1% accuracy for a direct test on an unseen independent dataset (OASIS). When directly testing this AD classifier on brain images of mild cognitive impairment (MCI) patients, 64.2% who finally converted into AD were predicted as AD, versus 25.9% who did not convert into AD (during the ADNI data collection period, but might convert in the future) were predicted as AD. The AD classifier also achieved high specificity in direct testing on other brain disorder datasets. Occlusion tests showed that hypothalamus, superior vermis, thalamus, amygdala and limbic system areas played critical roles in predicting sex, and hippocampus, parahippocampal gyrus, putamen and insula were crucial for predicting AD. Finally, the transfer learning framework failed to achieve practical accuracy for psychiatric disorders, which remain open questions for future studies. We openly shared our preprocessed data, trained model, code and framework, as well as built an online predicting website (http://brainimagenet.org:8088) for whomever is interested in testing our classifier with brain imaging data from anybody and from any scanner.