LIUM Machine Translation Systems for WMT17 News Translation Task

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 288–295 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics LIUM Machine Translation Systems for WMT17 News Translation Task Mercedes García-Martínez, Ozan Caglayan†, Walid Aransa Adrien Bardet, Fethi Bougares, Loïc Barrault LIUM, University of Le Mans †ozancag@gmail.com FirstName.LastName@univ-lemans.fr Abstract This paper describes LIUM submis- sions to WMT17 News Translation Task for English↔German, English↔Turkish, English→Czech and English→Latvian language pairs. We train BPE-based at- tentive Neural Machine Translation sys- tems with and without factored outputs using the open source nmtpy framework. Competitive scores were obtained by en- sembling various systems and exploiting the availability of target monolingual cor- pora for back-translation. The impact of back-translation quantity and quality is also analyzed for English→Turkish where our post-deadline submission surpassed the best entry by +1.6 BLEU. 1 Introduction This paper describes LIUM Neural Machine Translation (NMT) submissions to WMT17 News Translation Task for English↔German, English↔Turkish, English→Czech and English→Latvian language pairs. We exper- imented with and without back-translation data for English↔German and English↔Turkish which are respectively described in Sections 3 and 4. For the latter pair, we also present an analysis about the impact of back-translation quality and quantity as well as two architectural ablations regarding the initialization and the output of recurrent decoder (Section 3). Experiments for English→Czech and English→Latvian are performed using Fac- tored NMT (FNMT) (García-Martínez et al., 2016) systems. FNMT is an extension of NMT which aims at simultaneously predicting the canonical form of a word and its morphological information needed to generate the final surface form. The details and results are presented in sec- tion 5. All submitted systems1 are trained using the open source nmtpy2 framework (Caglayan et al., 2017). 2 Baseline NMT Our baseline NMT is an attentive encoder-decoder (Bahdanau et al., 2014) implementation. A bi- directional Gated Recurrent Unit (GRU) (Chung et al., 2014) encoder is used to compute source sentence annotation vectors. We equipped the en- coder with layer normalization (Ba et al., 2016), a technique which adaptively normalizes the incom- ing activations of each hidden unit with a learnable gain and bias, after empirically observing that it improves both convergence speed and translation performance. A conditional GRU (CGRU) (Firat and Cho, 2016; Sennrich et al., 2017) decoder with atten- tion mechanism is used to generate a probability distribution over target tokens for each decoding step t. The hidden state of the CGRU is initialized using a non-linear transformation of the average encoder state produced by the encoder. Follow- ing Inan et al. (2016); Press and Wolf (2017), the feedback embeddings (input to the decoder) and the output embeddings are tied to enforce learn- ing a single target representation and decrease the number of total parameters by target vocabulary size × embedding size. We used Adam (Kingma and Ba, 2014) as the optimizer with a learning rate of 4e−4. Weights are initialized with Xavier scheme (Glorot and Bengio, 2010) and the total gradient norm is clipped to 5 (Pascanu et al., …

Content truncated to 500 words