Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions

Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions Marcin Junczys-Dowmunt1,2, Tomasz Dwojak1,2, Hieu Hoang3 1Faculty of Mathematics and Computer Science, Adam Mickiewicz University in Pozna´n 2School of Informatics, University of Edinburgh 3Moses Machine Translation CIC {junczys,t.dwojak}@amu.edu.pl hieu@hoang.co.uk Abstract In this paper we provide the largest published compar- ison of translation quality for phrase-based SMT and neural machine translation across 30 translation direc- tions. For ten directions we also include hierarchical phrase-based MT. Experiments are performed for the recently published United Nations Parallel Corpus v1.0 and its large six-way sentence-aligned subcorpus. In the second part of the paper we investigate aspects of trans- lation speed, introducing AmuNMT, our efficient neural machine translation decoder. We demonstrate that cur- rent neural machine translation could already be used for in-production systems when comparing words-per- second ratios. 1. Introduction We compare the performance of phrase-based SMT, hi- erarchical phrase-based, and neural machine translation (NMT) across fifteen language pairs and thirty transla- tion directions. [1] recently published the United Na- tions Parallel Corpus v1.0. It contains a subcorpus of ca. 11M sentences fully aligned across six languages (Arabic, Chinese, English, French, Russian, and Span- ish) and official development and test sets, which makes this an ideal resource for experiments across multiple language pairs. It is also a compelling use case for in-domain translation with large bilingual in-house re- sources. We provide BLEU scores for the entire trans- lation matrix for all official languages from the fully aligned subcorpus. We also introduce AmuNMT1, our efficient neural machine translation decoder and demonstrate that the current set-up could already be used instead of Moses 1https://github.com/emjotde/amunmt in terms of translation speed when a single GPU per machine is available. Multiple GPUs would surpass the speed of the proposed in-production Moses configura- tion by far. 2. Training data 2.1. The UN corpus The United Nations Parallel Corpus v1.0 [1] consists of human translated UN documents from the last 25 years (1990 to 2014) for the six official UN languages, Arabic, Chinese, English, French, Russian, and Span- ish. Apart from the pairwise aligned documents, a fully aligned subcorpus for the six official UN languages is distributed. This subcorpus consists of sentences that are consistently aligned across all languages with the English primary documents. Statistics for the data are provided below: Documents Lines English Tokens 86,307 11,365,709 334,953,817 Table 1: Statistics for fully aligned subcorpus Documents released in 2015 (excluded from the main corpus) were used to create official development and test sets for machine translation tasks. Develop- ment data was randomly selected from documents that were released in the first quarter of 2015, test data was selected from the second quarter. Both sets comprise 4,000 sentences that are 1-1 alignments across all official languages. As in the case of the fully aligned subcorpus, any translation direction can be evaluated. 2.2. Preprocessing Sentences longer than 100 words were discarded. We lowercased the training data as it was done in [1]; the data was tokenized with the Moses …

Content truncated to 500 words