On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4013–4023 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4013 On the Robustness of Unsupervised and Semi-supervised Cross-lingual Word Embedding Learning Yerai Doval*,1 Jose Camacho-Collados*,2 Luis Espinosa-Anke,2 Steven Schockaert2 1Grupo COLE, Escola Superior de Enxe˜nar´ıa Inform´atica, Universidade de Vigo, Spain yerai.doval@uvigo.es 2School of Computer Science and Informatics, Cardiff University, UK {camachocolladosj, espinosa-ankel, schockaerts1}@cardiff.ac.uk Abstract Cross-lingual word embeddings are vector representations of words in different languages where words with similar meaning are represented by similar vectors, regardless of the language. Recent developments which construct these embeddings by aligning monolingual spaces have shown that accurate alignments can be obtained with little or no supervision, which usually comes in the form of bilingual dictionaries. However, the focus has been on a particular controlled scenario for evaluation, and there is no strong evidence on how current state-of-the-art systems would fare with noisy text or for language pairs with major linguistic differences. In this paper we present an extensive evaluation over multiple cross-lingual embedding models, analyzing their strengths and limitations with respect to different variables such as target language, training corpora and amount of supervision. Our conclusions put in doubt the view that high-quality cross-lingual embeddings can always be learned without much supervision. Keywords: Multilinguality, Evaluation Methodologies, Semantics, Semi-supervised, weakly-supervised and unsupervised learn- ing 1. Introduction The standard approach for training word embeddings is to rely on monolingual corpora, which means in particular that a separate embedding model is learned for each language. There is an increasing interest, however, in learning cross- lingual word embeddings, where words from different lan- guages are mapped onto a single space. Such representa- tions are attractive, for instance, for dealing with the multi- lingual nature of text on the Web, but also as a vehicle for transferring knowledge (e.g., labelled training data) from resource-rich languages such as English to other languages (Ruder et al., 2018). Initially, the main obstacle to learning such cross-lingual embeddings was the need for large multilingual parallel corpora (Klementiev et al., 2012; Chandar et al., 2014; Lu- ong et al., 2015). This limitation, however, was alleviated by the development of methods that only need compara- ble data (e.g., Wikipedia corpora in different languages) as the main source of supervision (Vuli´c and Moens, 2015). In a complementary direction, it has recently been shown that high-quality cross-lingual embeddings can be obtained by aligning two independently learned monolingual embed- ding spaces. This strategy is appealing, as it means that one only needs access to monolingual corpora and a bilingual dictionary as supervision signal. Surprisingly, perhaps, it turns out that dictionaries with less than 100 word pairs are sufficient to obtain good alignments (Artetxe et al., 2017). In fact, recent works have shown that cross-lingual embed- dings can even be learned without any user-provided dic- tionary (Conneau et al., 2018a; Artetxe et al., 2018b; Xu et al., 2018). Despite the promising results reported in the literature, it re- mains unclear under …

Content truncated to 500 words