Modeling sparse longitudinal data on Riemannian manifolds