Evaluating Sub-word Embeddings in Cross-lingual Models

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2712–2719 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2712 Evaluating Sub-word embeddings in cross-lingual models Ali Hakimi Parizi, Paul Cook University of New Brunswick Fredericton, New Brunswick, Canada {ahakimi, paul.cook,}@unb.ca Abstract Cross-lingual word embeddings create a shared space for embeddings in two languages, and enable knowledge to be transferred between languages for tasks such as bilingual lexicon induction. One problem, however, is out-of-vocabulary (OOV) words, for which no embeddings are available. This is particularly problematic for low-resource and morphologically-rich languages, which often have relatively high OOV rates. Approaches to learning sub-word embeddings have been proposed to address the problem of OOV words, but most prior work has not considered sub-word embeddings in cross-lingual models. In this paper, we consider whether sub-word embeddings can be leveraged to form cross-lingual embeddings for OOV words. Specifically, we consider a novel bilingual lexicon induction task focused on OOV words, for language pairs covering several language families. Our results indicate that cross-lingual representations for OOV words can indeed be formed from sub-word embeddings, including in the case of a truly low-resource morphologically-rich language. Keywords: Cross-lingual Word Embeddings, Low-resource Languages, Morphologically-rich Languages 1. Introduction Cross-lingual word embeddings provide a shared space for embeddings in two languages, enabling knowledge to be transferred between them. Cross-lingual word embeddings can be used for tasks such as bilingual lexicon induction, and can be leveraged to improve systems for natural language processing (NLP) for low-resource languages for tasks such as language modelling (Adams et al., 2017), part-of-speech tagging (Fang and Cohn, 2017), and dependency parsing (Duong et al., 2015). In the case of out-of-vocabulary (OOV) words, however, no information is available. This could be particularly problematic for low-resource languages, where the number of words that embeddings are learned for could be relatively low due to the relatively small amount of training data available, and for morphologically rich languages, where many wordforms would not be observed while learning the embeddings. Sub-word level embeddings (Bojanowski et al., 2017, e.g.) — i.e., embeddings for units smaller than words, such as character sequences — have been proposed to address this limitation concerning OOV words for monolingual embedding models, but little prior work — with the notable exception of (Braune et al., 2018) — has considered sub-word embeddings in cross-lingual models. In this paper we evaluate whether sub-word embeddings can be leveraged in cross-lingual models. Specifically, we consider a novel bilingual lexicon induction task in which an in-vocabulary target language translation is found for an OOV source language word, where the representation of the source language word is constructed from sub- word embeddings. Our findings indicate that sub-word embeddings do carry information that can be leveraged in cross-lingual models. 2. Related Work A variety of approaches have been proposed to find cross- lingual embedding spaces. Some of these methods require very expensive signals for training, such as sentence alignments (Zou et al., 2013, e.g.) or comparable documents …

Content truncated to 500 words