Improving Neural Machine Translation Using Noisy Parallel Data through Distillation

Improving Neural Machine Translation Using Noisy Parallel Data through Distillation Praveen Dakwale Informatics Institute University of Amsterdam p.dakwale@uva.nl Christof Monz Informatics Institute University of Amsterdam c.monz@uva.nl Abstract Due to the scarcity of parallel training data for many language pairs, quasi-parallel or comparable training data provides an important alternative resource for train- ing machine translation systems for such language pairs. Since comparable cor- pora are not of as high quality as man- ually annotated parallel data, using them for training can have a negative effect on the translation performance of an NMT model. We propose distillation as a remedy to effectively leverage comparable data where the training of a student model on combined clean and comparable data is guided by a teacher model trained on the high-quality, clean data only. Our experiments for Arabic-English, Chinese- English, and German-English translation demonstrate that distillation yields signif- icant improvements compared to off-the- shelf use of comparable data and performs comparable to state-of-the-art methods for noise filtering. 1 Introduction Traditional machine translation systems are trained on parallel corpora consisting of sentences in the source language aligned to their translations in the target language. However, for many language pairs substantial amounts of high-quality parallel cor- pora are not available. On the other hand, for many languages, another useful resource known as com- parable corpora can be obtained relatively easily c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. in substantially larger amounts. Such comparable corpora can be created by crawling large monolin- gual data in the source and target languages from multilingual news portals such as Agence France- Presse (AFP), BBC news, Euronews etc. Source and target sentences in these monolingual corpora are then aligned by automatic document and sen- tence alignment techniques (Munteanu and Marcu, 2005). Such a bitext extracted from comparable data is usually not of the same quality as annotated parallel corpora. Recent research has shown that building models from low-quality data can have a degrading effect on the performance of recur- rent NMT models (Khayrallah and Koehn, 2018). Therefore, there is a growing interest in filtering and sampling techniques to extract high-quality sentence pairs from such large noisy parallel texts. Recently, the “Parallel corpus filtering” (Koehn et al., 2018) shared task was held at WMT-2018. This task aims at extracting high-quality sentence pairs from Paracrawl1, which is a large noisy par- allel corpus. Most of the participants in this task, used rule-based pre-filtering followed by a classifier-based scoring of sentence pairs (Barbu and Barbu Mititelu, 2018; Junczys-Dowmunt, 2018; Hangya and Fraser, 2018). A subset sam- pled with a fixed number of target tokens is then used to train recurrent NMT systems in order to evaluate the relative quality of the filtered bitexts. Some of the submissions show good translation performance for the German-English translation task by training on the filtered bitext only. In this paper, we propose a strategy to leverage additional low-quality bitexts without any filtering when used in conjunction …

Content truncated to 500 words