Mutation, selection, and ancestry in branching models: a variational approach

We consider the evolution of populations under the joint action of mutation and differential reproduction, or selection. The population is modelled as a finite-type Markov branching process in continuous time, and the associated genealogical tree is viewed both in the forward and the backward direction of time. The stationary type distribution of the reversed process, the so-called ancestral distribution, turns out as a key for the study of mutation–selection balance. This balance can be expressed in the form of a variational principle that quantifies the respective roles of reproduction and mutation for any possible type distribution. It shows that the mean growth rate of the population results from a competition for a maximal long-term growth rate, as given by the difference between the current mean reproduction rate, and an asymptotic decay rate related to the mutation process; this tradeoff is won by the ancestral distribution. We then focus on the case when the type is determined by a sequence of letters (like nucleotides or matches/mismatches relative to a reference sequence), and we ask how much of the above competition can still be seen by observing only the letter composition (as given by the frequencies of the various letters within the sequence). If mutation and reproduction rates can be approximated in a smooth way, the fitness of letter compositions resulting from the interplay of reproduction and mutation is determined in the limit as the number of sequence sites tends to infinity. Our main application is the quasispecies model of sequence evolution with mutation coupled to reproduction but independent across sites, and a fitness function that is invariant under permutation of sites. In this model, the fitness of letter compositions is worked out explicitly. In certain cases, their competition leads to a phase transition.