Population dynamics in sideband cooling of trapped ions outside the Lamb-Dicke regime