Monolingual Embeddings for Low Resourced Neural Machine Translation

Monolingual Embeddings for Low Resourced Neural Machine Translation Mattia Antonino Di Gangi1,2, Marcello Federico1 1Fondazione Bruno Kessler, Trento, Italy 2ICT Doctoral School - University of Trento, Italy {digangi,federico}@fbk.eu Abstract Neural machine translation (NMT) is the state of the art for machine translation, and it shows the best performance when there is a considerable amount of data available. When only little data exist for a language pair, the model cannot produce good representations for words, particularly for rare words. One common solution consists in reducing data sparsity by segmenting words into sub-words, in order to allow rare words to have shared representations with other words. Tak- ing a different approach, in this paper we present a method to feed an NMT network with word embeddings trained on monolingual data, which are combined with the task-specific embeddings learned at training time. This method can lever- age an embedding matrix with a huge number of words, which can therefore extend the word-level vocabulary. Our experiments on two language pairs show good results for the typical low-resourced data scenario (IWSLT in-domain dataset). Our consistent improvements over the baselines represent a positive proof about the possibility to leverage models pre-trained on monolingual data in NMT. 1. Introduction Neural machine translation [1, 2] has shown to be highly ef- fective in conditions where there is a good quantity of data available, but struggles to provide good results in a low- resource condition. In general, publicly-available parallel data are small in size, containing at most only few millions of parallel sentences. Therefore, it becomes important to in- crease the quantity of data by using monolingual data, which are always available in a larger quantity. Improving MT with monolingual data is a long-standing technique from statistical machine translation (SMT) [3]. In that case, target-side monolingual data are used to train a better language model for producing more fluent transla- tions [4], or even to perform domain adaptation [5]. By con- trast, there are no effective usages of source-side monolin- gual data. In NMT, there is only one model trained end to end instead of several different statistical models that are combined by means of a log-linear function. The end-to-end approach is considered to be the strength point of NMT [6], but it also means that there is no obvious way to use monolingual data. In fact, the most used approach so far consists in augmenting the training set with synthetic parallel data. They are usu- ally back-translations of target monolingual sentences [7], but also forward-translations of the source side [8] or even copies of the target language in the source side [9]. In all the cases, as the synthetic data are mixed with the real data, the number of synthetic sentence pairs should be kept un- der control to prevent a degradation of performance. This strongly limits the size of usable monolingual data. Other ap- proaches explore different machine learning frameworks for using monolingual data, such as multi-task learning [10] to improve the encoder with source-side monolingual data …

Content truncated to 500 words