Quantifying Numerical and Spatial Reliability of Amygdala and Hippocampal Subdivisions in FreeSurfer

On-going, large-scale neuroimaging initiatives have produced many MRI datasets with hundreds, even thousands, of individual participants and scans. These databases can aid in uncovering neurobiological causes and correlates of poor mental health, disease pathology, and many other important factors. While volumetric quantification of brain structures can be completed by expert hand-tracing, automated segmentations are becoming the only truly tractable approach for particularly large datasets. Here, we assessed the spatial and numerical reliability for newly-deployed automated segmentation of hippocampal subfields and amygdala nuclei in FreeSurfer (v7.1.0). In a sample of participants with repeated structural imaging scans (N=118), we found numerical reliability (as assessed by intraclass correlations, ICC) was reasonable, with an average ICCs of 0.853 for hippocampal subfields, and 0.878 for amygdala subnuclei. However, only 26% of all subfields and subnuclei were “excellent” in terms of having numerical reliability ≥ 0.90. Spatial reliability was similarly reasonable, with 39% of hippocampal subfields and 33% of amygdala subnuclei having Dice coefficients ≥ 0.70. However, and of note, multiple regions (e.g., portions of the Dentate Gyrus; Cornu Ammonis 3; Medial and Paralaminar amygdala nuclei) had poor numerical and/or spatial reliability. We also examined correlations between spatial reliability and person-level factors (e.g., participant age and sex; inter-scan interval, and difference in image quality). For these factors, inter-scan interval and sex were related to variations in spatial reliability metrics. Examined collectively, our work suggests caution should be exercised for a few hippocampal subfields and amygdala nuclei with more variable reliability.

[1]  A. Dale,et al.  High‐resolution intersubject averaging and a coordinate system for the cortical surface , 1999, Human brain mapping.

[2]  Benoit M. Dawant,et al.  Morphometric analysis of white matter lesions in MR images: method and validation , 1994, IEEE Trans. Medical Imaging.

[3]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[4]  Bruce Fischl,et al.  Highly accurate inverse consistent registration: A robust approach , 2010, NeuroImage.

[5]  Lin Shi,et al.  Using Large-Scale Statistical Chinese Brain Template (Chinese2020) in Popular Neuroimage Analysis Toolkits , 2017, Front. Hum. Neurosci..

[6]  S Robinson,et al.  Optimized 3 T EPI of the amygdalae , 2004, NeuroImage.

[7]  Bing Chen,et al.  An open science resource for establishing reliability and reproducibility in functional connectomics , 2014, Scientific Data.

[8]  D. Altman,et al.  STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[9]  M. Jenkinson,et al.  Hippocampal volume across age: Nomograms derived from over 19,700 people in UK Biobank , 2019, NeuroImage: Clinical.

[10]  J. Kramer,et al.  Comparing Volume Loss in Neuroanatomical Regions of Emotion versus Regions of Cognition in Healthy Aging , 2016, PloS one.

[11]  M. Mallar Chakravarty,et al.  Quantitative comparison of 21 protocols for labeling hippocampal subfields and parahippocampal subregions in in vivo MRI: Towards a harmonized segmentation protocol , 2015, NeuroImage.

[12]  Michael Davis,et al.  The amygdala , 2000, Current Biology.

[13]  Diane E. Stodola,et al.  Preschool Externalizing Behavior Predicts Gender-Specific Variation in Adolescent Neural Structure , 2015, PloS one.

[14]  Martin Styner,et al.  A comparison of automated segmentation and manual tracing for quantifying hippocampal and amygdala volumes , 2009, NeuroImage.

[15]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[16]  Andrew L. Alexander,et al.  A 3D Fully Convolutional Neural Network With Top-Down Attention-Guided Refinement for Accurate and Robust Automatic Segmentation of Amygdala and Its Subnuclei , 2020, Frontiers in Neuroscience.

[17]  L Cipolotti,et al.  A volumetric study of hippocampus and amygdala in depressed patients with subjective memory problems. , 2000, The Journal of neuropsychiatry and clinical neurosciences.

[18]  Jamie L. Hanson,et al.  Behavioral Problems After Early Life Stress: Contributions of the Hippocampus and Amygdala , 2015, Biological Psychiatry.

[19]  F. Pestilli.  Human white matter and knowledge representation , 2018, PLoS biology.

[20]  Anders M. Dale,et al.  MRI-derived measurements of human subcortical, ventricular and intracranial brain volumes: Reliability effects of scan sessions, acquisition sequences, data analyses, scanner upgrade, scanner vendors and field strengths , 2009, NeuroImage.

[21]  J. Whitwell,et al.  Alzheimer's disease neuroimaging , 2018, Current opinion in neurology.

[22]  Valerie A. Carr,et al.  Hippocampal subfield volumetry from structural isotropic 1 mm3 MRI scans: A note of caution , 2020, Human brain mapping.

[23]  C. Jack,et al.  Brain atrophy rates predict subsequent clinical conversion in normal elderly and amnestic MCI , 2005, Neurology.

[24]  James J. Knierim,et al.  CA3 Retrieves Coherent Representations from Degraded Input: Direct Evidence for CA3 Pattern Completion and Dentate Gyrus Pattern Separation , 2014, Neuron.

[25]  Jia Liu,et al.  A test-retest dataset for assessing long-term reliability of brain morphology and resting-state brain activity , 2016, Scientific Data.

[26]  M. Mallar Chakravarty,et al.  Hippocampus and amygdala volumes from magnetic resonance images in children: Assessing accuracy of FreeSurfer and FSL against manual segmentation , 2016, NeuroImage.

[27]  Bruce Fischl,et al.  Within-subject template estimation for unbiased longitudinal image analysis , 2012, NeuroImage.

[28]  Koenraad Van Leemput,et al.  Bayesian longitudinal segmentation of hippocampal substructures in brain MRI using subject-specific atlases , 2016, NeuroImage.

[29]  Bruce Fischl,et al.  FreeSurfer , 2012, NeuroImage.

[30]  Eric Achten,et al.  Intra- and interobserver variability of MRI-based volume measurements of the hippocampus and amygdala using the manual ray-tracing method , 1998, Neuroradiology.

[31]  André J. W. van der Kouwe,et al.  Reliability of MRI-derived cortical and subcortical morphometric measures: Effects of pulse sequence, voxel geometry, and parallel imaging , 2009, NeuroImage.

[32]  Paul A. Yushkevich,et al.  Progress update from the hippocampal subfields group , 2019, Alzheimer's & dementia.

[33]  K. Konrad,et al.  Accuracy and bias of automatic hippocampal segmentation in children and adolescents , 2018, Brain Structure and Function.

[34]  Ewald Moser,et al.  On the origin of respiratory artifacts in BOLD-EPI of the human brain. , 2002, Magnetic resonance imaging.

[35]  Koenraad Van Leemput,et al.  A computational atlas of the hippocampal formation using ex vivo, ultra-high resolution MRI: Application to adaptive segmentation of in vivo MRI , 2015, NeuroImage.

[36]  Glenda M MacQueen,et al.  Course of illness, hippocampal function, and hippocampal volume in major depression , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[37]  Anders M. Dale,et al.  Sequence-independent segmentation of magnetic resonance images , 2004, NeuroImage.

[38]  A. M. Dale,et al.  A hybrid approach to the skull stripping problem in MRI , 2004, NeuroImage.

[39]  Andrew R. Bender,et al.  Age differences in hippocampal subfield volumes from childhood to late adulthood , 2016, Hippocampus.

[40]  M. Alda,et al.  Bilateral hippocampal volume increases after long-term lithium treatment in patients with bipolar disorder: a longitudinal MRI study , 2007, Psychopharmacology.

[41]  A. Hedayat,et al.  Statistical Methods in Assessing Agreement , 2002 .

[42]  Chunfeng Lian,et al.  A Longitudinal MRI Study of Amygdala and Hippocampal Subfields for Infants with Risk of Autism , 2019, GLMI@MICCAI.

[43]  Alysha Gilmore,et al.  Variations in Structural MRI Quality Significantly Impact Commonly-Used Measures of Brain Anatomy , 2020 .

[44]  David H. Salat,et al.  Test-retest reliability of FreeSurfer automated hippocampal subfield segmentation within and across scanners , 2020, NeuroImage.

[45]  J. MacKillop,et al.  Adverse Childhood Experiences and Amygdalar Reduction: High-Resolution Segmentation Reveals Associations With Subnuclei and Psychiatric Outcomes , 2019, Child maltreatment.

[46]  Yushan Huang,et al.  In vivo quantification of amygdala subnuclei using 4.7 T fast spin echo imaging , 2017, NeuroImage.

[47]  Z. M. Saygina,et al.  High-resolution magnetic resonance imaging reveals nuclei of the human amygdala : manual segmentation to automatic atlas , 2017 .

[48]  Valerie A. Carr,et al.  A harmonized segmentation protocol for hippocampal and parahippocampal subregions: Why do we need one and what are the key goals? , 2017, Hippocampus.

[49]  Bradford C. Dickerson,et al.  A reliable protocol for the manual segmentation of the human amygdala and its subregions using ultra-high resolution MRI , 2012, NeuroImage.

[50]  D. Cui,et al.  Study on the sub-regions volume of hippocampus and amygdala in schizophrenia. , 2019, Quantitative imaging in medicine and surgery.

[51]  R W Cox,et al.  AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. , 1996, Computers and biomedical research, an international journal.

[52]  Christopher R. Madan,et al.  Test–retest reliability of brain morphology estimates , 2017, Brain Informatics.

[53]  F Andermann,et al.  Anatomic basis of amygdaloid and hippocampal volume measurement by magnetic resonance imaging , 1992, Neurology.

[54]  Nicholas J. Buser,et al.  Variations in Structural MRI Quality Significantly Impact Commonly-Used Measures of Brain Anatomy , 2019, bioRxiv.

[55]  Giovanni B. Frisoni,et al.  Brain morphometry reproducibility in multi-center 3T MRI studies: A comparison of cross-sectional and longitudinal segmentations , 2013, NeuroImage.

[56]  Jozsef Janszky,et al.  Are there any gender differences in the hippocampus volume after head-size correction? A volumetric and voxel-based morphometric study , 2014, Neuroscience Letters.

[57]  Gregory McCarthy,et al.  Scan–rescan reliability of subcortical brain volumes derived from automated segmentation , 2010, Human brain mapping.

[58]  Mitul A Mehta,et al.  Test–retest reliability and longitudinal analysis of automated hippocampal subregion volumes in healthy ageing and Alzheimer's disease populations , 2018, Human brain mapping.

[59]  Armin von Gunten,et al.  Hippocampal volume and subjective memory impairment in depressed patients , 2004, European Psychiatry.

[60]  P. Myles,et al.  Using the Bland-Altman method to measure agreement with repeated measures. , 2007, British journal of anaesthesia.

[61]  Yong He,et al.  A connectivity-based test-retest dataset of multi-modal magnetic resonance imaging in young healthy adults , 2015, Scientific Data.

[62]  Ron Kikinis,et al.  Statistical validation of image segmentation quality based on a spatial overlap index. , 2004, Academic radiology.

[63]  Chen Sun,et al.  Distinct Neural Circuits for the Formation and Retrieval of Episodic Memories , 2017, Cell.

[64]  Anders M. Dale,et al.  A hybrid approach to the Skull Stripping problem in MRI , 2001, NeuroImage.

[65]  A. Dale,et al.  Cortical Surface-Based Analysis II: Inflation, Flattening, and a Surface-Based Coordinate System , 1999, NeuroImage.

[66]  Andrew J. Saykin,et al.  The open diffusion data derivatives, brain data upcycling via integrated publishing of derivatives and reproducible open cloud services , 2018, Scientific Data.

[67]  Anders M. Dale,et al.  Cortical Surface-Based Analysis I. Segmentation and Surface Reconstruction , 1999, NeuroImage.

[68]  A. Dale,et al.  Whole Brain Segmentation Automated Labeling of Neuroanatomical Structures in the Human Brain , 2002, Neuron.

[69]  Susumu Tonegawa,et al.  Conjunctive input processing drives feature selectivity in hippocampal CA1 neurons , 2015, Nature Neuroscience.

[70]  Qiyong Gong,et al.  Comparison of the brain development trajectory between Chinese and U.S. children and adolescents , 2015, Front. Syst. Neurosci..

[71]  Jens Frahm,et al.  COMMENTS AND CONTROVERSIES Functional MRI of the Human Amygdala , 2001 .

[72]  Ingrid Agartz,et al.  Neuroimaging hippocampal subfields in schizophrenia and bipolar disorder: A systematic review and meta-analysis. , 2018, Journal of psychiatric research.

[73]  Paul M. Thompson,et al.  Heritability and reliability of automatically segmented human hippocampal formation subregions , 2015, NeuroImage.

[74]  H. Wagner,et al.  Amygdala Nuclei Volume and Shape in Military Veterans With Posttraumatic Stress Disorder. , 2019, Biological psychiatry. Cognitive neuroscience and neuroimaging.

[75]  Alois Schlögl,et al.  Synaptic mechanisms of pattern completion in the hippocampal CA3 network , 2016, Science.

[76]  Dhruv Marwha,et al.  Meta-analysis reveals a lack of sexual dimorphism in human amygdala volume , 2017, NeuroImage.

[77]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal Chiropractic Medicine.

[78]  Tom Johnstone,et al.  Amygdala Volume and Nonverbal Social Impairment in Adolescent and Adult Males with Autism , 2022 .

[79]  Denis Dooley,et al.  Atlas of the Human Brain. , 1971 .

[80]  Nick C Fox,et al.  The Alzheimer's disease neuroimaging initiative (ADNI): MRI methods , 2008, Journal of magnetic resonance imaging : JMRI.

[81]  Daniel P. Kennedy,et al.  The Autism Brain Imaging Data Exchange: Towards Large-Scale Evaluation of the Intrinsic Brain Architecture in Autism , 2013, Molecular Psychiatry.

[82]  Brian B. Avants,et al.  Robust Automated Amygdala Segmentation via Multi-Atlas Diffeomorphic Registration , 2012, Front. Neurosci..

[83]  Richard Camicioli,et al.  Aging hippocampus and amygdala , 2008, Neuroreport.

[84]  Lutz Jäncke,et al.  Reliability and statistical power analysis of cortical and subcortical FreeSurfer metrics in a large sample of healthy elderly , 2015, NeuroImage.

[85]  Oliver T Wolf,et al.  MRI volume of the amygdala: a reliable method allowing separation from the hippocampal formation , 1999, Psychiatry Research: Neuroimaging.