A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 765–774, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments Omer Levy∗ University of Washington Seattle, WA omerlevy@gmail.com Anders Søgaard∗ University of Copenhagen Copenhagen, Denmark soegaard@di.ku.dk Yoav Goldberg Bar-Ilan University Ramat-Gan, Israel yoav.goldberg@gmail.com Abstract While cross-lingual word embeddings have been studied extensively in recent years, the qualitative differences between the different algorithms remain vague. We observe that whether or not an algorithm uses a particular feature set (sentence IDs) accounts for a significant performance gap among these algorithms. This feature set is also used by traditional alignment al- gorithms, such as IBM Model-1, which demonstrate similar performance to state- of-the-art embedding algorithms on a va- riety of benchmarks. Overall, we observe that different algorithmic approaches for utilizing the sentence ID feature space re- sult in similar performance. This paper draws both empirical and theoretical par- allels between the embedding and align- ment literature, and suggests that adding additional sources of information, which go beyond the traditional signal of bilin- gual sentence-aligned corpora, may sub- stantially improve cross-lingual word em- beddings, and that future baselines should at least take such features into account. 1 Introduction Cross-lingual word embedding algorithms try to represent the vocabularies of two or more lan- guages in one common continuous vector space. These vectors can be used to improve monolingual word similarity (Faruqui and Dyer, 2014) or sup- port cross-lingual transfer (Gouws and Søgaard, 2015). In this work, we focus on the sec- ond (cross-lingual) aspect of these embeddings, and try to determine what makes some embed- ding approaches better than others on a set of ∗These authors contributed equally to this work. translation-oriented benchmarks. While cross- lingual word embeddings have been used for a va- riety of cross-lingual transfer tasks, we prefer eval- uating on translation-oriented benchmarks, rather than across specific NLP tasks, since the transla- tion setting allows for a cleaner examination of cross-lingual similarity. Another important de- lineation of this work is that we focus on algo- rithms that rely on sentence-aligned data; in part, because these algorithms are particularly interest- ing for low-resource languages, but also to make our analysis and comparison with alignment algo- rithms more focused. We observe that the top performing embed- ding algorithms share the same underlying fea- ture space – sentence IDs – while their different algorithmic approaches seem to have a negligi- ble impact on performance. We also notice that several statistical alignment algorithms, such as IBM Model-1 (Brown et al., 1993), operate un- der the same data assumptions. Specifically, we find that using the translation probabilities learnt by Model-1 as the cross-lingual similarity func- tion (in place of the commonly-used cosine simi- larity between word embeddings) performs on-par with state-of-the-art cross-lingual embeddings on word alignment and bilingual dictionary induction tasks. In other words, as long as the similarity function is based on the sentence …

Content truncated to 500 words