Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency