The AFRL WMT17 Neural Machine Translation Training Task Submission

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 687–691 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics The AFRL WMT17 Neural Machine Translation Training Task Submission Grant Erdmann, Katherine Young, Jeremy Gwinnup Air Force Research Laboratory grant.erdmann@us.af.mil, katherinemccreightyoung@gmail.com, jeremy.gwinnup.1@us.af.mil Abstract The WMT17 Neural Machine Translation Training Task aims to test various methods of training neural machine translation sys- tems. We describe the AFRL submission, including preprocessing and its knowledge distillation framework. Teacher systems are given factors for domain, case, and subword location. Student systems are given multiple teachers’ output and a sub- selected set of the training data designed to match the target domain. Numerical re- sults indicate that the student systems sur- pass the teachers in translation quality and that this benefit comes directly from the in- clusion of the teachers’ output. 1 Introduction This paper describes our development of sys- tems for the WMT17 Neural Machine Translation (NMT) Training Task (WMT, 2017). This task tests methods of adjusting the NMT training pro- cess, with a fixed size and format for the final English-to-Czech system. A large (approx. 50 mil- lion line) general-domain (mostly subtitles) bilin- gual corpus is provided as a training set. A domain is provided for each line of this corpus. News text, the application domain, composes about 0.5% of the corpus (see Table 1, column “Given”). A sub- word expansion to be used is explicitly provided as well. We preprocess the training data to standard- ize some punctuation and character encoding dif- ferences. We filter the data to remove some lines of foreign languages and little information, approxi- mately 5% of the training data. We follow a teacher-student (aka knowledge distillation) paradigm for this task (Ba and Caru- ana, 2014). We train ten replicate systems larger than the final system, based on all the training data available. These systems are aware of different factors (domain, case, subword location) for each subword, allowing them to use this information to learn finer details of translation. They also produce different outputs, based on randomness in train- ing. We translate the entire news-domain train- ing corpus with all replicate systems. These out- puts are added to the most applicable training data as another set of references, and the final NMT systems are trained from this decimated and aug- mented training set. We choose to resist making many changes to the given systems, in order to provide useful a poste- riori comparisons. To this end, we use: • only neuralmonkey, or branches thereof, for NMT • the given data only • alterations to given 4GB and 8GB configura- tions only. for all intermediate systems. 2 Preprocessing of Training Data 2.1 Normalization We use several simple steps of text normaliza- tion to produce a more standardized training set. Some lines had been doubly-tokenized, and we correct these (e.g., “& quot ;” becomes “"”). Punctuation and spacing indicators are made uniform (e.g., “Tha \x09D s” becomes “That ' s”). Several characters were denoted by …

Content truncated to 500 words