Towards Handling Compositionality in Low-Resource Bilingual Word Induction

Towards Handling Compositionality in Low-Resource Bilingual Word Induction Viktor Hangya hangyav@cis.lmu.de Alexander Fraser fraser@cis.lmu.de Center for Information and Language Processing, LMU Munich, Munich, Germany Abstract Bilingual word embeddings (BWEs) facilitate the translation of single source language words to single target language words. However, often a single source word must be translated using two target words. Previous approaches depend on observing the two target language words as a (frequent) bigram in a corpus. But for many languages only a small amount of written text is available, so that such “atomic” embeddings can only be built for a small number of frequent bigrams. In this paper, we extend atomic embedding based approaches to improve the 1-to-2 word translation of rare words by decomposing the representation of a source word to represen- tations of two target words, allowing to model translations for which the required bigram was not observed in our monolingual corpora. We create a gold standard lexicon for 1-to-2 transla- tion containing source German compounds along with their translations to two English words, and show that our approach improves performance. We also show the importance of bigrams for the downstream task of unsupervised machine translation and show small but significant BLEU score improvements with our approach. Our approach is an important first step in the direction of handling composition in BWEs, beyond simple memorization of seen bigrams. 1 Introduction Bilingual word embeddings (BWEs) are key components in cross-lingual NLP tasks alleviating data scarcity for many languages. They can be built using source and target language mono- lingual corpora with either a cheap bilingual signal (Mikolov et al., 2013b; Xing et al., 2015) or no bilingual signal at all (Conneau et al., 2018; Artetxe et al., 2018a). They are applied to many downstream tasks, such as bilingual lexicon induction (BLI) (Vuli´c and Korhonen, 2016) and cross-lingual transfer learning (Schuster et al., 2019). Unsupervised machine translation (UMT) strongly depends on BWEs. Meaningful translations can be generated without any bilingual signal by using BWEs to translate words 1-to-1. However, many words in a given language are the composition of multiple smaller lexi- cal units which are expressed individually in other languages, such as the German compound Waschmaschine → washing machine. Using the idea of atomic embeddings for frequent bi- grams (Mikolov et al., 2013c) previous work proposed 1-to-2 word translations achieving sig- nificant improvements in UMT (Lample et al., 2018b; Artetxe et al., 2018b). These approaches start by learning individual vector representations for frequent bigrams using their monolingual contexts, effectively treating them as if they were single words. After the projection of the learned monolingual spaces to BWEs, these special bigrams can also be translated or serve as translation candidates in exactly the same way as unigrams. On the other hand, these approaches need to learn these special atomic embeddings in advance, requiring them to be frequently ob- served to be learned well. This assumes the availability of large monolingual corpora which are Proceedings of the 14th Conference of the Association for Machine Translation …

Content truncated to 500 words