Fitness costs associated with ancestry to isolated populations of an endangered species