Prefix Language Models are Unified Modal Learners