Joint Training for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora

Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics (*SEM), pages 39–49 Barcelona, Spain (Online), December 12–13, 2020 39 Joint Training for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora Ali Hakimi Parizi Faculty of Computer Science University of New Brunswick ahakimi@unb.ca Paul Cook Faculty of Computer Science University of New Brunswick paul.cook@unb.ca Abstract In this paper, we propose a novel method for learning cross-lingual word embeddings, that incorporates sub-word information dur- ing training, and is able to learn high-quality embeddings from modest amounts of mono- lingual data and a bilingual lexicon. This method could be particularly well-suited to learning cross-lingual embeddings for lower- resource, morphologically-rich languages, en- abling knowledge to be transferred from rich- to lower-resource languages. We evaluate our proposed approach simulating lower-resource languages for bilingual lexicon induction, monolingual word similarity, and document classification. Our results indicate that incor- porating sub-word information indeed leads to improvements, and in the case of document classification, performance better than, or on par with, strong benchmark approaches. 1 Introduction State-of-the-art approaches in natural language processing (NLP) typically require a substantial amount of human-annotated data (i.e, for super- vised approaches to tasks such as part-of-speech tagging or dependency parsing) or they need a very large amount of unannotated text for train- ing (e.g., methods for learning word embeddings). This poses a particular problem for building NLP systems for low-resource languages. There are thousands of human languages, and creating an- notated datasets for all of them would be very ex- pensive. Furthermore, many languages have a rela- tively small number of speakers, and in many cases large amounts of text are not readily-available for building corpora for these languages. A further related challenge is posed by morphologically-rich This work is licensed under a Creative Commons Attri- bution 4.0 International License. License details: http: //creativecommons.org/licenses/by/4.0/. languages, because many word-forms would not be expected to be observed in a training corpus. One way to address these problems is to trans- fer knowledge from a rich-resource language to a lower-resource language. Word Embeddings are a key feature in ap- proaches for a wide range of NLP tasks, such as part-of-speech tagging (Al-Rfou’ et al., 2013), de- pendency parsing (Chen and Manning, 2014), and named entity recognition (Pennington et al., 2014). If we are able to transfer the knowledge captured in word embeddings for a rich-resource language to another low-resource language, then develop- ing NLP tools could become more feasible for the low-resource language. There has therefore been a wealth of research on cross-lingual word em- beddings (e.g., Mikolov et al., 2013b; Vuli´c and Moens, 2016; Lample et al., 2018), in which em- beddings for multiple languages are learned in a shared space, and which can be used to transfer knowledge between languages, such as from a rich- resource language to a low-resource one (Ruder et al., 2019). Despite the wide range of research on learn- ing cross-lingual embeddings, there are some lim- itations of these methods that have not been ad- dressed. In the …

Content truncated to 500 words