The University of Edinburgh’s Neural MT Systems for WMT17

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 389–399 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics The University of Edinburgh’s Neural MT Systems for WMT17 Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone and Philip Williams University of Edinburgh, Scotland Abstract This paper describes the University of Edinburgh’s submissions to the WMT17 shared news translation and biomedical translation tasks. We participated in 12 translation directions for news, translating between English and Czech, German, Lat- vian, Russian, Turkish and Chinese. For the biomedical task we submitted systems for English to Czech, German, Polish and Romanian. Our systems are neural ma- chine translation systems trained with Ne- matus, an attentional encoder-decoder. We follow our setup from last year and build BPE-based models with parallel and back- translated monolingual training data. Nov- elties this year include the use of deep ar- chitectures, layer normalization, and more compact models due to weight tying and improvements in BPE segmentations. We perform extensive ablative experiments, reporting on the effectivenes of layer nor- malization, deep architectures, and differ- ent ensembling techniques. 1 Introduction We participated in the WMT17 shared news trans- lation task for 12 translation directions, translat- ing between English and Czech, German, Latvian, Russian, Turkish and Chinese, and in the WMT17 shared biomedical translation task for English to Czech, German, Polish and Romanian.1 We sub- mitted neural machine translation systems trained with Nematus (Sennrich et al., 2017). Our setup is based on techniques described in last year’s sys- tem description (Sennrich et al., 2016a), includ- ing the use of subword models (Sennrich et al., 1 We provide trained models and training commands at http://data.statmt.org/wmt17_systems/ 2016c), back-translated monolingual data, (Sen- nrich et al., 2016b), and re-ranking with right-to- left models. This year, we experimented with deep network architectures, new ways to include monolingual data, and different ensembling variants. Other novelties include obtaining more compact models via better BPE segmentation and by weight tying (Press and Wolf, 2017), and speeding up model training with layer normalization (Ba et al., 2016) and adam (Kingma and Ba, 2015). We perform extensive ablative experiments across language pairs to evaluate the effectiveness of each of these approaches. When comparing this year’s baseline models to our best results, we show consistent increases in scores of 2.2–5 BLEU for our 12 news task language pairs. Among the con- strained submissions to the news task, our submis- sions are ranked tied first for 11 out of the 12 trans- lation directions in which we participated. For the biomedical task, we obtained the highest BLEU for all our submitted systems. For the 6 language pairs for which we partici- pated both in WMT16 and WMT17, we also show the scores of last year’s systems. We observe solid improvements with increases of 1.5–3 BLEU for single models. Some of these improvements are due to differences in training data, preprocessing and hyperparameters, but most of the increase is due to …

Content truncated to 500 words