Reconsidering Cross-lingual Word Embeddings

While cross-lingual word embeddings have been studied exte nsiv ly in recent years, the qualitative differences between the different algorithms remains vague. We observe that whether or not an algorithm uses a particular feature set (sentence IDs) ac counts for a significant performance gap among these algorithms. This feature set is also used by t raditional alignment algorithms, such as IBM Model-1, which demonstrate similar performance to state-of-the-art embedding algorithms on a variety of benchmarks. Overall, we observe tha t different algorithmic approaches for utilizing the sentence ID feature space result in simila r performance. This paper draws both empirical and theoretical parallels between the embedding and alignment literature, and suggests that adding additional sources of information, which go beyond the traditional signal of bilingual sentence-aligned corpora, is an appealing appro ach for substantially improving crosslingual word embeddings.