Do Pre-trained Models Benefit Equally in Continual Learning?