Flow-history effect on higher modes in the spherical Couette system