Fetal brain age estimation and anomaly detection using attention-based deep ensembles with uncertainty

MRI-based brain age prediction has been widely used to characterize normal brain development, and deviations from the typical developmental trajectory are indications of brain abnormalities. Age prediction of the fetal brain remains unexplored, although it can be of broad interest to prenatal examination given the limited diagnostic tools available for assessment of the fetal brain. In this work, we built an attention-based deep residual network based on routine clinical T2-weighted MR images of 659 fetal brains, which achieved an overall mean absolute error of 0.767 weeks and R2 of 0.920 in fetal brain age prediction. The predictive uncertainty and estimation confidence were simultaneously quantified from the network as markers for detecting fetal brain anomalies using an ensemble method. The novel markers overcame the limitations in conventional brain age estimation and demonstrated promising diagnostic power in differentiating several types of fetal abnormalities, including small head circumference, malformations and ventriculomegaly with the area under the curve of 0.90, 0.90 and 0.67, respectively. In addition, attention maps were derived from the network, which revealed regional features that contributed to fetal age estimation at each gestational stage. The proposed attention-based deep ensembles approach demonstrated superior performance in fetal brain age estimation and fetal anomaly detection, which has the potential to be translated to prenatal diagnosis in clinical practice.

[1]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[2]  Nassir Navab,et al.  Bayesian QuickNAT: Model uncertainty in deep whole-brain segmentation for structure-wise quality control , 2018, NeuroImage.

[3]  Liyue Shen,et al.  Deep Learning with Attention to Predict Gestational Age of the Fetal Brain , 2018, ArXiv.

[4]  Karolinska Schizophrenia,et al.  Common brain disorders are associated with heritable patterns of apparent aging of the brain , 2019 .

[5]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[6]  Hai-Gwo Hwu,et al.  Generalization of diffusion magnetic resonance imaging–based brain age prediction model through transfer learning , 2020, NeuroImage.

[7]  Eileen Luders,et al.  Brain maturation: Predicting individual BrainAGE in children and adolescents using structural MRI , 2012, NeuroImage.

[8]  J. Cole,et al.  Predicting Age Using Neuroimaging: Innovative Brain Ageing Biomarkers , 2017, Trends in Neurosciences.

[9]  Gang Li,et al.  Hierarchical Rough-to-Fine Model for Infant Age Prediction Based on Cortical Features , 2020, IEEE Journal of Biomedical and Health Informatics.

[10]  Daniela Prayer,et al.  Fetal MRI: techniques and protocols , 2004, Pediatric Radiology.

[11]  Gang Li,et al.  Multi-Branch Deformable Convolutional Neural Network with Label Distribution Learning for Fetal Brain Age Prediction , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[12]  Wiro J Niessen,et al.  Gray Matter Age Prediction as a Biomarker for Risk of Dementia , 2019, Proceedings of the National Academy of Sciences.

[13]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[14]  Jun Zhang,et al.  The research implications of the selection of a gestational age estimation method. , 2007, Paediatric and perinatal epidemiology.

[15]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[16]  Colin Studholme,et al.  Registration-based approach for reconstruction of high-resolution in utero fetal MR brain images. , 2006, Academic radiology.

[17]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Thomas E. Nichols,et al.  Quantifying uncertainty in brain-predicted age using scalar-on-image quantile regression , 2019, NeuroImage.

[19]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[20]  Jian Yang,et al.  Occluded Pedestrian Detection Through Guided Attention in CNNs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Robert Leech,et al.  Prediction of brain age suggests accelerated atrophy after traumatic brain injury , 2015, Annals of neurology.

[22]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[23]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[24]  Stefan Klöppel,et al.  BrainAGE in Mild Cognitive Impaired Patients: Predicting the Conversion to Alzheimer’s Disease , 2013, PloS one.

[25]  Daniela Prayer,et al.  Methods of fetal MR: beyond T2-weighted imaging , 2006 .

[26]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[27]  Wei Xu,et al.  Look and Think Twice: Capturing Top-Down Visual Attention with Feedback Convolutional Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[28]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[29]  A J Barkovich,et al.  Magnetic resonance imaging of the fetal brain and spine: an increasingly important tool in prenatal diagnosis, part 1. , 2006, AJNR. American journal of neuroradiology.

[30]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Byoung-Tak Zhang,et al.  Multimodal Residual Learning for Visual QA , 2016, NIPS.

[32]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[33]  Wesley K. Thompson,et al.  A Nonlinear Simulation Framework Supports Adjusting for Age When Analyzing BrainAGE , 2018, bioRxiv.

[34]  Max Welling,et al.  Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors , 2016, ICML.

[35]  James A Hanley,et al.  Random measurement error and regression dilution bias , 2010, BMJ : British Medical Journal.

[36]  Christian Wachinger,et al.  Gaussian process uncertainty in age estimation as a measure of brain abnormality , 2018, NeuroImage.

[37]  Thomas E. Nichols,et al.  Quantifying uncertainty in brain-predicted age using scalar-on-image quantile regression , 2020, NeuroImage.

[38]  Vijay K. Venkatraman,et al.  Neuroanatomical Assessment of Biological Maturity , 2012, Current Biology.

[39]  Christos Davatzikos,et al.  Use of Fetal Magnetic Resonance Image Analysis and Machine Learning to Predict the Need for Postnatal Cerebrospinal Fluid Diversion in Fetal Ventriculomegaly , 2017, JAMA pediatrics.

[40]  Giovanni Montana,et al.  Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker , 2016, NeuroImage.

[41]  P. Ellen Grant,et al.  Sulcal pits and patterns in developing human brains , 2019, NeuroImage.

[42]  Thomas E. Nichols,et al.  Estimation of Brain Age Delta from Brain Imaging , 2019 .

[43]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[44]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[45]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[46]  A. James Barkovich,et al.  Malformations of cortical development: diagnostic accuracy of fetal MR imaging. , 2012, Radiology.

[47]  Daniel S. Margulies,et al.  Predicting brain-age from multimodal imaging data captures cognitive impairment , 2016, NeuroImage.

[48]  H. Stefánsson,et al.  Brain age prediction using deep learning uncovers associated sequence variants , 2019, Nature Communications.

[49]  Yong Xia,et al.  Attention Residual Learning for Skin Lesion Classification , 2019, IEEE Transactions on Medical Imaging.

[50]  Gustav Mårtensson,et al.  AVRA: Automatic visual ratings of atrophy from MRI images using recurrent convolutional neural networks , 2018, NeuroImage: Clinical.

[51]  Thomas G. Dietterich Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[52]  Anders M. Dale,et al.  Common brain disorders are associated with heritable patterns of apparent aging of the brain , 2019, Nature Neuroscience.

[53]  Ayush Singh,et al.  Deep Predictive Motion Tracking in Magnetic Resonance Imaging: Application to Fetal Imaging , 2019, IEEE Transactions on Medical Imaging.

[54]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Marleen de Bruijne,et al.  3D regression neural network for the quantification of enlarged perivascular spaces in brain MRI , 2018, Medical Image Anal..

[56]  Siegfried Wahl,et al.  Leveraging uncertainty information from deep neural networks for disease detection , 2016, Scientific Reports.

[57]  Andreas Nürnberger,et al.  The Power of Ensembles for Active Learning in Image Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Galia Avidan,et al.  From a deep learning model back to the brain—Identifying regional predictors and their relation to aging , 2020, Human brain mapping.

[59]  F Prefumo,et al.  Additional value of fetal magnetic resonance imaging in the prenatal diagnosis of central nervous system anomalies: a systematic review of the literature , 2014, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[60]  Antonio Criminisi,et al.  Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolution , 2017, MICCAI.

[61]  Wiepke Cahn,et al.  Accelerated Brain Aging in Schizophrenia: A Longitudinal Pattern Recognition Study. , 2016, The American journal of psychiatry.

[62]  Y. Inoue,et al.  An Acid–Base Controlled Molecular Switch. syn–anti Conformational Switching in a μ-oxo Bis(Iron Porphyrin) , 2003 .

[63]  Yan Wang,et al.  Mapping fetal brain development based on automated segmentation and 4D brain atlasing , 2020, Brain Structure and Function.

[64]  Xiaogang Wang,et al.  Residual Attention Network for Image Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  R. Marioni,et al.  Brain age and other bodily ‘ages’: implications for neuropsychiatry , 2018, Molecular Psychiatry.

[66]  Stefan Klöppel,et al.  Estimating the age of healthy subjects from T1-weighted MRI scans using kernel methods: Exploring the influence of various parameters , 2010, NeuroImage.

[67]  Antonio Criminisi,et al.  Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement , 2019, ArXiv.

[68]  Been Kim,et al.  Sanity Checks for Saliency Maps , 2018, NeurIPS.

[69]  C. Clouchoux,et al.  Assessment of MRI-Based Automated Fetal Cerebral Cortical Folding Measures in Prediction of Gestational Age in the Third Trimester , 2015, American Journal of Neuroradiology.

[70]  D. Lev,et al.  Fetal brain imaging: a comparison between magnetic resonance imaging and dedicated neurosonography , 2004, Ultrasound in obstetrics & gynecology : the official journal of the International Society of Ultrasound in Obstetrics and Gynecology.

[71]  P. Griffiths,et al.  Use of MRI in the diagnosis of fetal brain abnormalities in utero (MERIDIAN): a multicentre, prospective cohort study , 2017, The Lancet.

[72]  Colin Studholme,et al.  Intersection Based Motion Correction of Multislice MRI for 3-D in Utero Fetal Brain Image Formation , 2010, IEEE Transactions on Medical Imaging.

[73]  Olivier Potvin,et al.  A novel patch-based procedure for estimating brain age across adulthood , 2019, NeuroImage.