SegSRGAN: Super-resolution and segmentation using generative adversarial networks - Application to neonatal brain MRI

BACKGROUND AND OBJECTIVE One of the main issues in the analysis of clinical neonatal brain MRI is the low anisotropic resolution of the data. In most MRI analysis pipelines, data are first re-sampled using interpolation or single image super-resolution techniques and then segmented using (semi-)automated approaches. In other words, image reconstruction and segmentation are then performed separately. In this article, we propose a methodology and a software solution for carrying out simultaneously high-resolution reconstruction and segmentation of brain MRI data. METHODS Our strategy mainly relies on generative adversarial networks. The network architecture is described in detail. We provide information about its implementation, focusing on the most crucial technical points (whereas complementary details are given in a dedicated GitHub repository). We illustrate the behavior of the proposed method for cortex analysis from neonatal MR images. RESULTS The results of the method, evaluated quantitatively (Dice, peak signal-to-noise ratio, structural similarity, number of connected components) and qualitatively on a research dataset (dHCP) and a clinical one (Epirmex), emphasize the relevance of the approach, and its ability to take advantage of data-augmentation strategies. CONCLUSIONS Results emphasize the potential of our proposed method/software with respect to practical medical applications. The method is provided as a freely available software tool, which allows one to carry out his/her own experiments, and involve the method for the super-resolution reconstruction and segmentation of arbitrary cerebral structures from any MR image dataset.

[1]  Sébastien Ourselin,et al.  AdaPT: An adaptive preterm segmentation algorithm for neonatal brain MRI , 2013, NeuroImage.

[2]  J. Lefévre,et al.  Are Developmental Trajectories of Cortical Folding Comparable Between Cross-sectional Datasets of Fetuses and Preterm Newborns? , 2016, Cerebral cortex.

[3]  G. Fitzgerald,et al.  'I. , 2019, Australian journal of primary health.

[4]  Daniel Rueckert,et al.  Automatic Whole Brain MRI Segmentation of the Developing Neonatal Brain , 2014, IEEE Transactions on Medical Imaging.

[5]  Isabelle Bloch,et al.  An iterative multi-atlas patch-based approach for cortex segmentation from neonatal MRI , 2018, Comput. Medical Imaging Graph..

[6]  N. Marlow,et al.  Neurologic and developmental disability at six years of age after extremely preterm birth. , 2005, The New England journal of medicine.

[7]  F. Lazeyras,et al.  Mapping the early cortical folding process in the preterm newborn brain. , 2008, Cerebral cortex.

[8]  Andrew Melbourne,et al.  Cortical folding of the preterm brain: a longitudinal analysis of extremely preterm born neonates using spectral matching , 2016, Brain and behavior.

[9]  Daniel Rueckert,et al.  Automatic tissue and structural segmentation of neonatal brain MRI using Expectation-Maximization , 2012 .

[10]  Daniel Rueckert,et al.  A review on automatic fetal and neonatal brain MRI segmentation , 2017, NeuroImage.

[11]  Dinggang Shen,et al.  Automatic segmentation of neonatal images using convex optimization and coupled level sets , 2011, NeuroImage.

[12]  Hayit Greenspan,et al.  Super-Resolution in Medical Imaging , 2009, Comput. J..

[13]  François Goffinet,et al.  EPIPAGE 2: a preterm birth cohort in France in 2011 , 2014, BMC Pediatrics.

[14]  John H. Gilmore,et al.  Automatic segmentation of MR images of the developing newborn brain , 2005, Medical Image Anal..

[15]  Andrew Melbourne,et al.  Cortical folding of the preterm brain: a longitudinal analysis of extremely preterm born neonates using spectral matching , 2016, Brain and behavior.

[16]  Andrea Vedaldi,et al.  Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Tuo Zhao,et al.  Towards Understanding the Importance of Noise in Training Neural Networks , 2019, ICML 2019.

[18]  Camille Couprie,et al.  Semantic Segmentation using Adversarial Networks , 2016, NIPS 2016.

[19]  Narendra Ahuja,et al.  Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Alan C. Evans,et al.  NEOCIVET: Extraction of Cortical Surface and Analysis of Neonatal Gyrification Using a Modified CIVET Pipeline , 2015, MICCAI.

[21]  Virendra Mishra,et al.  Structural Development of Human Fetal and Preterm Brain Cortical Plate Based on Population-Averaged Templates. , 2016, Cerebral cortex.

[22]  Michel Barlaud,et al.  Deterministic edge-preserving regularization in computed imaging , 1997, IEEE Trans. Image Process..

[23]  Joseph V. Hajnal,et al.  Development of cortical microstructure in the preterm human brain , 2013, Proceedings of the National Academy of Sciences.

[24]  Feng Shi,et al.  Brain MRI super resolution using 3D deep densely connected neural networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[25]  A. Fanaroff,et al.  Outcomes of children of extremely low birthweight and gestational age in the 1990's. , 1999, Early human development.

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Daniel Rueckert,et al.  Automatic segmentation and reconstruction of the cortex from neonatal MRI , 2007, NeuroImage.

[28]  Dwarikanath Mahapatra,et al.  Skull Stripping of Neonatal Brain MRI: Using Prior Shape Information with Graph Cuts , 2012, Journal of Digital Imaging.

[29]  Mitko Veta,et al.  Adversarial Training and Dilated Convolutions for Brain MRI Segmentation , 2017, DLMIA/ML-CDS@MICCAI.

[30]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[31]  Nicolas Passat,et al.  Simultaneous Super-Resolution and Segmentation Using a Generative Adversarial Network: Application to Neonatal Brain MRI , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[32]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[33]  Daniel Rueckert,et al.  The influence of preterm birth on the developing thalamocortical connectome , 2013, Cortex.

[34]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[35]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[36]  A. Fanaroff,et al.  Outcomes of children of extremely low birthweight and gestational age in the 1990s. , 2000, Seminars in neonatology : SN.

[37]  Dinggang Shen,et al.  4D Multi-Modality Tissue Segmentation of Serial Infant Images , 2012, PloS one.

[38]  Paul F. Whelan,et al.  A Hybrid Approach to Brain Extraction from Premature Infant MRI , 2011, SCIA.

[39]  Ahmed Serag,et al.  Accurate Learning with Few Atlases (ALFA): an algorithm for MRI neonatal brain extraction and comparison with 11 publicly available methods , 2016, Scientific Reports.

[40]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[41]  Chi-Hieu Pham,et al.  Brain MRI super-resolution using deep 3D convolutional networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[42]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  D. Louis Collins,et al.  Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation , 2011, NeuroImage.

[44]  Sergey Dolenko,et al.  Adding Noise During Training as a Method to Increase Resilience of Neural Network Solution of Inverse Problems: Test on the Data of Magnetotelluric Sounding Problem , 2017 .

[45]  Nicolas Passat,et al.  Multiscale brain MRI super-resolution using deep 3D convolutional networks , 2019, Comput. Medical Imaging Graph..