Improving Automatic Speech Recognition and Speech Translation via Word Embedding Prediction

In this article, we target speech translation (ST). We propose lightweight approaches that generally improve either ASR or end-to-end ST models. We leverage continuous representations of words, known as word embeddings, to improve ASR in cascaded systems as well as end-to-end ST models. The benefit of using word embedding is that word embedding can be obtained easily by training on pure textual data, which alleviates data scarcity issue. Also, word embedding provides additional contextual information to speech models. We motivate to distill the knowledge from word embedding into speech models. In ASR, we use word embeddings as a regularizer to reduce the WER, and further propose a novel decoding method to fuse the semantic relations among words for further improvement. In the end-to-end ST model, we propose leveraging word embeddings as an intermediate representation to enhance translation performance. Our analysis shows that it is possible to map speech signals to semantic space, which motivates future work on applying the proposed methods in spoken language processing tasks.