Should All Cross-Lingual Embeddings Speak English?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8658–8679 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8658 Should All Cross-Lingual Embeddings Speak English? Antonios Anastasopoulos and Graham Neubig Language Technologies Institute, Carnegie Mellon University {aanastas,gneubig}@cs.cmu.edu Abstract Most of recent work in cross-lingual word em- beddings is severely Anglocentric. The vast majority of lexicon induction evaluation dic- tionaries are between English and another lan- guage, and the English embedding space is se- lected by default as the hub when learning in a multilingual setting. With this work, how- ever, we challenge these practices. First, we show that the choice of hub language can sig- nificantly impact downstream lexicon induc- tion and zero-shot POS tagging performance. Second, we both expand a standard English- centered evaluation dictionary collection to in- clude all language pairs using triangulation, and create new dictionaries for under-represented languages.1 Evaluating established methods over all these language pairs sheds light into their suitability for aligning embeddings from distant languages and presents new challenges for the field. Finally, in our analysis we iden- tify general guidelines for strong cross-lingual embedding baselines, that extend to language pairs that do not include English. 1 Introduction Continuous vectors for representing words (embed- dings) (Turian et al., 2010) have become ubiquitous in modern, neural NLP. Cross-lingual representations (Mikolov et al., 2013) additionally represent words from various languages in a shared continuous space, which in turn can be used for Bilingual Lexicon Induction (BLI). BLI is often the first step towards several down- stream tasks such as Part-Of-Speech (POS) tagging (Zhang et al., 2016), parsing (Ammar et al., 2016a), document classification (Klementiev et al., 2012), and machine translation (Irvine and Callison-Burch, 2013; Artetxe et al., 2018b; Lample et al., 2018). Often, such shared representations are learned with a two-step process, whether under bilingual or multilin- gual settings (hereinafter BWE and MWE, respectively). First, monolingual word embeddings are learned over 1Available at https://github.com/antonisa/ embeddings. large swaths of text. Such pre-trained word embed- dings, such as the fastText Wikipedia vectors (Grave et al., 2018), are available for many languages and are widely used. Second, a mapping between the languages is learned in one of three ways: in a supervised manner if dictionaries or parallel data are available to be used for supervision (Zou et al., 2013), under minimal su- pervision e.g. using only identical strings (Smith et al., 2017), or even in an unsupervised fashion (Zhang et al., 2017; Conneau et al., 2018). Both in bilingual and mul- tilingual settings, it is common that one of the language embedding spaces is the target to which all other lan- guages get aligned (hereinafter “the hub"). We outline the details in Section 2. Despite all the recent progress in learning cross- lingual embeddings, we identify a major shortcoming to previous work: it is by and large English-centric. Notably, most MWE approaches essentially select En- glish as the hub during training by default, aligning all other language spaces to the English one. We …

Content truncated to 500 words