Optimization and comparative evaluation of nonlinear deformation algorithms for atlas-based segmentation of DBS target nuclei

&NA; Nonlinear registration of individual brain MRI scans to standard brain templates is common practice in neuroimaging and multiple registration algorithms have been developed and refined over the last 20 years. However, little has been done to quantitatively compare the available algorithms and much of that work has exclusively focused on cortical structures given their importance in the fMRI literature. In contrast, for clinical applications such as functional neurosurgery and deep brain stimulation (DBS), proper alignment of subcortical structures between template and individual space is important. This allows for atlas‐based segmentations of anatomical DBS targets such as the subthalamic nucleus (STN) and internal pallidum (GPi). Here, we systematically evaluated the performance of six modern and established algorithms on subcortical normalization and segmentation results by calculating over 11,000 nonlinear warps in over 100 subjects. For each algorithm, we evaluated its performance using T1‐or T2‐weighted acquisitions alone or a combination of T1‐, T2‐and PD‐weighted acquisitions in parallel. Furthermore, we present optimized parameters for the best performing algorithms. We tested each algorithm on two datasets, a state‐of‐the‐art MRI cohort of young subjects and a cohort of subjects age‐ and MR‐quality‐matched to a typical DBS Parkinson's Disease cohort. Our final pipeline is able to segment DBS targets with precision comparable to manual expert segmentations in both cohorts. Although the present study focuses on the two prominent DBS targets, STN and GPi, these methods may extend to other small subcortical structures like thalamic nuclei or the nucleus accumbens. HighlightsWe compared six nonlinear deformation algorithms for atlas‐based segmentation of two common DBS targets, the STN and GPi.Parameters of the best performing algorithms were optimized to maximize precision of subcortical deformations.These optimized pipelines are able to segment DBS targets with precision that is comparable to manual expert segmentations.The optimized pipelines have been made available to the scientific community through the open source toolbox Lead‐DBS.