Relative performance of Bayesian clustering software for inferring population substructure and individual assignment at low levels of population differentiation

Traditional methods for characterizing genetic differentiation among populations rely on a priori grouping of individuals. Bayesian clustering methods avoid this limitation by using linkage and Hardy–Weinberg disequilibrium to decompose a sample of individuals into genetically distinct groups. There are several software programs available for Bayesian clustering analyses, all of which describe a decrease in the ability to detect distinct clusters as levels of genetic differentiation among populations decrease. However, no study has yet compared the performance of such methods at low levels of population differentiation, which may be common in species where populations have experienced recent separation or high levels of gene flow. We used simulated data to evaluate the performance of three Bayesian clustering software programs, PARTITION, STRUCTURE, and BAPS, at levels of population differentiation below FST=0.1. PARTITION was unable to correctly identify the number of subpopulations until levels of FST reached around 0.09. Both STRUCTURE and BAPS performed very well at low levels of population differentiation, and were able to correctly identify the number of subpopulations at FST around 0.03. The average proportion of an individual’s genome assigned to its true population of origin increased with increasing FST for both programs, reaching over 92% at an FST of 0.05. The average number of misassignments (assignments to the incorrect subpopulation) continued to decrease as FST increased, and when FST was 0.05, fewer than 3% of individuals were misassigned using either program. Both STRUCTURE and BAPS worked extremely well for inferring the number of clusters when clusters were not well-differentiated (FST=0.02–0.03), but our results suggest that FST must be at least 0.05 to reach an assignment accuracy of greater than 97%.