Copied Monolingual Data Improves Low-Resource Neural Machine Translation

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 148–156 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Copied Monolingual Data Improves Low-Resource Neural Machine Translation Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield School of Informatics, University of Edinburgh a.currey@sms.ed.ac.uk {amiceli, kheafiel}@inf.ed.ac.uk Abstract We train a neural machine translation (NMT) system to both translate source- language text and copy target-language text, thereby exploiting monolingual cor- pora in the target language. Specifically, we create a bitext from the monolingual text in the target language so that each source sentence is identical to the tar- get sentence. This copied data is then mixed with the parallel corpus and the NMT system is trained like normal, with no metadata to distinguish the two input languages. Our proposed method proves to be an effective way of incorporating monolingual data into low-resource NMT. On Turkish↔English and Romanian↔English translation tasks, we see gains of up to 1.2 BLEU over a strong baseline with back-translation. Further analysis shows that the linguis- tic phenomena behind these gains are different from and largely orthogonal to back-translation, with our copied corpus method improving accuracy on named entities and other words that should remain identical between the source and target languages. 1 Introduction Neural machine translation (NMT) systems re- quire a large amount of training data to make generalizations, both on the source side (in or- der to interpret the text well enough to translate it) and on the target side (in order to produce flu- ent translations). This data typically comes in the form of parallel corpora, in which each sentence in the source language is matched to a transla- tion in the target language. Recent work (Gul- cehre et al., 2015; Sennrich et al., 2016b) has investigated incorporating monolingual training data (particularly on the target side) into NMT. This effectively converts machine translation into a semi-supervised problem that takes advantage of both labeled (parallel) and unlabeled (mono- lingual) data. Adding monolingual data to NMT is important because sufficient parallel data is un- available for all but a few language pairs and do- mains. In this paper, we introduce a straightforward method for adding target-side monolingual train- ing data to an NMT system without changing its architecture or training algorithm. This method converts a monolingual corpus in the target lan- guage into a parallel corpus by copying it, so that each source sentence is identical to its correspond- ing target sentence. This copied corpus is then mixed with the original parallel data and used to train the NMT system, with no distinction made between the parallel and the copied data. We focus on language pairs with small amounts of parallel data where monolingual data has the most impact. On the relatively low- resource language pairs of English↔Turkish and English↔Romanian, we find that our copying technique is effective both alone and combined with back-translation. This is the case even when no additional monolingual data is used (i.e. when the copied corpus and the …

Content truncated to 500 words