Where do the Progenitors of Millisecond Pulsars come from