Genealogy and subpopulation differentiation under various models of population structure

Abstract. The structured coalescent is used to calculate some quantities relating to the genealogy of a pair of homologous genes and to the degree of subpopulation differentiation, under a range of models of subdivided populations and assuming the infinite alleles model of neutral mutation. The classical island and stepping-stone models of population structure are considered, as well as two less symmetric models. For each model, we calculate the Laplace transform of the distribution of the coalescence time of a pair of genes from specified locations and the corresponding mean and variance. These results are then used to calculate the values of Wright’s coefficient FST, its limit as the mutation rate tends to zero and the limit of its derivative with respect to the mutation rate as the mutation rate tends to zero. From this derivative it is seen that FST can depend strongly on the mutation rate, for example in the case of an essentially one-dimensional habitat with many subpopulations where gene flow is restricted to neighbouring subpopulations.